nems.db module

nems.db.Engine()[source]

Returns a mysql engine object. Creates the engine if necessary. Otherwise returns the existing one.

nems.db.Session()[source]

Returns a mysql session object.

nems.db.Tables()[source]

Returns a dictionary containing Narf database table objects.

nems.db.sqlite_test()[source]
nems.db._get_db_uri()[source]

Used by Engine() to establish a connection to the database.

nems.db.pd_query(sql=None, params=None)[source]

execture an SQL command and return the results in a dataframe params:

sql: string

query to execute use fprintf formatting, eg

sql = “SELECT * FROM table WHERE name=%s” params = (“Joe”)

TODO : sqlite compatibility?

nems.db.sql_command(sql)[source]
nems.db.enqueue_models(celllist, batch, modellist, force_rerun=False, user='nems', codeHash='master', jerbQuery='', executable_path=None, script_path=None, priority=1, GPU_job=0, reserve_gb=0)[source]

Call enqueue_single_model for every combination of cellid and modelname contained in the user’s selections.

for each cellid in celllist and modelname in modellist, will create jobs that execute this command on a cluster machine:

<executable_path> <script_path> <cellid> <batch> <modelname>

e.g.: /home/nems/anaconda3/bin/python /home/nems/nems/fit_single_model.py TAR010c-18-1 271 ozgf100ch18_dlog_wcg18x1_stp1_fir1x15_lvl1_dexp1_basic

celllistlist

List of cellid selections made by user.

batchstring

batch number selected by user.

modellistlist

List of modelname selections made by user.

force_rerunboolean (default=False)

If true, models will be fit even if a result already exists. If false, models with existing results will be skipped.

userstring (default=”nems”)

Typically the login name of the user who owns the job

codeHashstring (default=”master”)

Git hash string identifying a commit for the specific version of the code repository that should be used to run the model fit. Can also accept the name of a branch.

jerbQuerydict

Dict that will be used by ‘jerb find’ to locate matching jerbs

executable_pathstring (defaults to nems’ python3 executable)

Path to executable python (or other command line program)

script_pathstring (defaults to nems’ copy of nems/nems_fit_single.py)

First parameter to pass to executable

GPU_job: int

Whether or not to run run as a GPU job.

(queueids, messages)list

Returns a tuple of the tQueue id and results message for each job that was either updated in or added to the queue.

Narf_Analysis : enqueue_models_callback

nems.db.enqueue_single_model(cellid, batch, modelname, user=None, force_rerun=False, codeHash='master', jerbQuery='', executable_path=None, script_path=None, priority=1, GPU_job=0, reserve_gb=0)[source]
nems.db.add_job_to_queue(args, note, force_rerun=False, user='nems', codeHash='master', jerbQuery='', executable_path=None, script_path=None, priority=1, GPU_job=0, reserve_gb=0)[source]

Low level interaction with tQueue to run single generic job on cluster

<executable_path> <script_path> <arg1> <arg2> <arg3> …

args: list of system arguments to be passed to script_path note: unique id for this job ex: “Pupil job: AMT004b12_p_TOR” force_rerun : boolean (default=False)

If true, models will be fit even if a result already exists. If false, models with existing results will be skipped.

userstring (default=”nems”)

Typically the login name of the user who owns the job

codeHashstring (default=”master”)

Git hash string identifying a commit for the specific version of the code repository that should be used to run the model fit. Can also accept the name of a branch.

jerbQuerydict

Dict that will be used by ‘jerb find’ to locate matching jerbs

executable_pathstring (defaults to nems’ python3 executable)

Path to executable python (or other command line program)

script_pathstring (defaults to nems’ copy of nems/nems_fit_single.py)

First parameter to pass to executable

(queueids, messages)list

Returns a tuple of the tQueue id and results message for each job that was either updated in or added to the queue.

Narf_Analysis : enqueue_models_callback

nems.db._add_model_to_queue(commandPrompt, note, user, codeHash, jerbQuery, priority=1, rundataid=0)[source]
jobtQueue object instance

tQueue object with variables assigned inside function based on arguments.

Narf_Analysis: dbaddqueuemaster

nems.db.update_job_complete(queueid=None)[source]

mark job queueid complete in tQueue svd old-fashioned way of doing it

nems.db.update_job_start(queueid=None)[source]

in tQueue, mark job as active and progress set to 1

nems.db.update_startdate(queueid=None)[source]

in tQueue, update the PID

nems.db.update_job_pid(pid, queueid=None)[source]

in tQueue, update the PID

nems.db.update_job_tick(queueid=None)[source]

update current machine’s load in the cluster db and tick off a step of progress in the fit in tQueue

nems.db.save_results(modelspec, preview_file, queueid=None)[source]

save performance data from modelspec to Results pull some information out of the queue table if queueid provided

nems.db.update_results_table(modelspec, preview=None, username='svd', labgroup='lbhb', public=1)[source]

Save information about a fit based on modelspec.meta :param modelspec: NEMS modelspec :param preview: filename of saved results figure (optional) :param username: username id for logging :param labgroup: labgroup id for logging :param public: (True) if True, flag as publicly visible outside of labgroup :return: results_id identifier for new/updated entry in Results table

nems.db.fetch_meta_data(modelspec, r, attrs, cellid=None)[source]

Assign attributes from model fitter object to Results object.

modelspecnems modelspec with populated metadata dictionary

Stack containing meta data, modules, module names et cetera (see nems_modules).

rsqlalchemy ORM object instance

Results object, either a blank one that was created before calling this function or one that was retrieved via a query to Results.

Nothing. Attributes of ‘r’ are modified in-place.

nems.db._fetch_attr_value(modelspec, k, default=0.0, cellid=None)[source]

Return the value of key ‘k’ of modelspec.meta, or default.

nems.db.get_batch(name=None, batchid=None)[source]
nems.db.add_batch_data(cellid, batch, recording_uri_list)[source]

Save information about a fit based on modelspec.meta :param cellid: :param batch: :param recording_uri_list: :return:

nems.db.get_batch_cells(batch=None, cellid=None, rawid=None, as_list=False)[source]

Retrieve a dataframe from Data containing all cellids in a batch.

batchint

The batch number to retrieve cellids from.

cellidstr

A full or partial (must include beginning) cellid to match entries to. Ex: cellid=’AMT’ would return all cellids beginning with AMT.

rawidint

A full rawid to match entries to (must be an exact match).

as_listbool

If true, return cellids as a list instead of a dataframe. (default False).

dInteger-indexed dataframe with one column for matched cellids and

one column for batch number. If as_list=True, d will instead be a list of cellids.

nems.db.get_batch_cell_data(batch=None, cellid=None, rawid=None, label=None)[source]
nems.db.get_batches(name=None)[source]
nems.db.get_cell_files(cellid=None, runclass=None, rawid=None)[source]
nems.db.get_cell_files2(cellid=None, runclass=None, rawid=None)[source]
nems.db.get_isolation(cellid=None, batch=None)[source]
nems.db.get_cellids(rawid=None)[source]
nems.db.list_batches(name=None)[source]
nems.db.get_data_parms(rawid=None, parmfile=None)[source]
nems.db.get_siteid(s)[source]

strip off the tail of a cellid to get the siteid

nems.db.get_batch_modelnames(batch, cellid_filter=None, modelname_filter=None, min_count=0)[source]
nems.db.get_batch_sites(batch, modelname_filter=None)[source]

get all siteids and a representative cellid from each site in a batch :param batch: NEMS batch :return: (siteids, cellids) tuple lists of siteids and cellids

nems.db.batch_comp(batch=301, modelnames=None, cellids=None, stat='r_test', join_how='left')[source]
nems.db.get_results_file(batch, modelnames=None, cellids=None)[source]
nems.db.export_fits(batch, modelnames=None, cellids=None, dest=None)[source]
Parameters
  • batch – required

  • modelnames – required - [list of modelnames]

  • cellids – [list of cellids] - default all cells

  • dest – path where exported models should be saved

Returns

nems.db.get_stable_batch_cells(batch=None, cellid=None, rawid=None, label='parm')[source]
New functionality as of 03/13/2020
Previous: returned set of cellids that was stable across all files

e.g. if 20 cells were recorded across four files but only 15 of the cells were isolated across all four files, this would return a list of only those 15 cells and the rawid for all four files.

New behavior: Now, does the opposite of before and returns the 20 cellids

and only the three file rawids where all 20 cells were stable.

If rawid is passed, this will only return the cells that were stable across those rawids. So, in the example, if you passed all four rawids you’d only get the 15 cellids back corresponding to the stable isolation units.

nems.db.get_wft(cellid=None)[source]
nems.db.get_gSingleCell_meta(cellid=None, fields=None)[source]
nems.db.get_rawid(cellid, run_num)[source]

Used to return the rawid corresponding to given run number. To be used if you have two files at a given site that belong to the same batch but were sorted separately and you only want to load cellids from one of the files.

ex. usage in practice would be to pass a sys arg cellid followed by the run_num:

cellid = ‘TAR017b-04-1_04’

This specifies cellid and run_num. So parse this string and pass as args to this function to return rawid

nems.db.get_pen_location(cellid)[source]

Cellid can be string or list. For every channel in the list, return the well position. For example, if cellid = [‘AMT024a-01-2’, ‘AMT024a-03-2’] then this code expects there to be at least 3 well positions in the db, it will return the 0th and 2nd positions.

DO NOT pass list of cellids from different sites. This will not work

If recording with a single probe array, there is only one well position for all 64 channels. For this reason, it doesn’t make sense for cellid to be a list

nems.db.save_recording_to_db(recfilepath, meta=None, user='nems', labgroup='', public=True)[source]

expects recfilepath == “/path/to/data/<exptname>_<hash>.tgz”

nems.db.recent_models(batch, N=20)[source]