Getting Started with Socraticqs2 Development¶

Installation and setup¶

Recommended: use virtualenv to create an isolated test environment¶

I find it convenient to isolate my development environment from any other Python installation, by using virtualenv. To create an empty virtualenv (replace the path with whatever you want):

virtualenv /path/to/put/your/new/ve

To start working with this virtualenv in a given terminal session (either to install pip packages into it as below, or to actually use it), type (again replace the path with what you used in the previous step):

source /path/to/put/your/new/ve/bin/activate

Pre-requisites¶

To run the Socraticqs2 development/test server on your computer, you need various things such as:

Python 2.7
Git: we use Git as our version control; if you don’t already have it, you can download either Github for Windows or Github for Mac; most linux package managers can install Git for you automatically. See further details below.
PostgreSQL: back-end database for Socraticqs2. See further details below.
Pandoc (currently used for converting text to HTML): you can download this for Mac OS X; most linux package managers can install it automatically.
Python packages automatically installable by pip using our dev_requirements.txt requirements file: e.g. Django, pypandoc, django-crispy-forms, wikipedia etc. This is described in detail below.

Warning

On Ubuntu-like distros, you must first install some dependencies of the python lxml package, before running pip install below:

sudo apt-get install libxml2-dev libxslt1-dev python-dev zlib1g-dev

Installing Git version control software¶

We use Git and GitHub for working on Socraticqs2 development. You’ll first need to get Git:

for Windows and Mac, you can download it from the Git website (command-line interface, recommended). We also recommend installing a graphical interface for viewing Git code revision history such as SourceTree.
Alternatively, you can download the GitHub for Windows / Mac app from GitHub (more limited, in our humble opinion).
On Linux, your package manager can install (and update) Git for you automatically. We recommend installing both the command line tools (git) and a graphical interface (gitg).

If you are not accustomed to using Git / GitHub as your primary version control system, we suggest you review our Git Usage Guide.

Getting the Socraticqs2 source code¶

To get your own copy of the Socraticqs2 code, first go to our Github repository https://github.com/cjlee112/socraticqs2 and Fork the repository (you’ll need a Github account to do this). This creates your own repository in your Github account, which you can make changes to, and issue pull requests for us to incorporate your changes. Next clone the repository to your local computer, one of two ways:

via the command line: click the “HTTPS clone URL” copy to clipboard button on your GitHub repository page, and paste it into a terminal command like so (substitute your correct clone URL):
```
git clone https://github.com/YOURNAME/socraticqs2.git
```
This will clone the repository to a new directory socraticqs2/ in your current directory. We recommend you also add our main repository as a “remote” repository called upstream:
```
cd socraticqs2
git remote add upstream https://github.com/cjlee112/socraticqs2.git
```
via Github for Windows or Mac: you can just click the Clone to Desktop link on the webpage for your fork (repository).

Installing and configuring PostgreSQL¶

Ubuntu 14.04 distribution instructions¶

You can check PostgreSQL installation manual for linux on official PostgreSQL page.

Steps to install PostgreSQL on Ubuntu machine:

Create the file /etc/apt/sources.list.d/pgdg.list, and add a line for the repository
deb http://apt.postgresql.org/pub/repos/apt/ trusty-pgdg main

Import the repository signing key, and update the package lists

wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | \
  sudo apt-key add -
sudo apt-get update

Install postgres-9.4:
```
sudo apt-get install postgresql-9.4
```

Warning

On Ubuntu-like distros, you must first install postgresql-server-dev-9.4 as a dependency to python psycopg2 package, before running pip install below:

sudo apt-get install postgresql-server-dev-9.4

Next we need to set default postgres user password to start development process:
```
sudo -u postgres psql postgres
postgres=# \password postgres
```
Now we can exit from PostgreSQL and start to configure our Django project to use created database:
```
postgres=# \q
```
But one more thing we need to do is to open pg_hba.conf file:
```
sudo vim /etc/postgresql/9.4/main/pg_hba.conf
```

And change:

local   all    postgres    peer

to:

local   all    postgres    md5

Finally we need to restart PostgreSQL server:
```
sudo service postgresql restart
```

In case when you don’t want to have the PostgreSQL server running all the time on your computer, just to play with the Socraticqs2 code (e.g. this consumes system resources), you can start and stop PostgreSQL manually:

sudo service postgresql stop
sudo service postgresql start

There are couple of ways to prevent PostgreSQL to start on system boot:

First one is to remove system start-up links for postgres:

✗ sudo update-rc.d -f postgresql remove
  Removing any system startup links for /etc/init.d/postgresql ...
    /etc/rc0.d/K21postgresql
    /etc/rc1.d/K21postgresql
    /etc/rc2.d/S19postgresql
    /etc/rc3.d/S19postgresql
    /etc/rc4.d/S19postgresql
    /etc/rc5.d/S19postgresql
    /etc/rc6.d/K21postgresql

In this case we can use default way to start postgres manually:
sudo service postgresql start

Second one is to change line auto to manual in /etc/postgresql/9.4/main/start.conf:

# Automatic startup configuration
# auto: automatically start/stop the cluster in the init script
# manual: do not start/stop in init scripts, but allow manual startup with
#         pg_ctlcluster
# disabled: do not allow manual startup with pg_ctlcluster (this can be easily
#           circumvented and is only meant to be a small protection for
#           accidents).

auto

In this case we prevent to start main postgres cluster and need to use next commands to stop|start pg_cluster:
sudo pg_ctlcluster 9.4 main stop
sudo pg_ctlcluster 9.4 main start

More information you can find on PostgreSQL 9.4 documentation page.

Mac OS X installation¶

For installing PostgreSQL on Mac OS X please follow the official instructions.

One of the prefered way to install PostreSQL is Homebrew:

brew install postgres

Postgres will be installed into /usr/local/var/postgres directory.

Information about how to start Postgres you can find by following command:

brew info postgres

By default Homebrew create postgres role with name as your mac username.

So you need to set a password for this role:

psql postgres
\password your_username

Configuring Django project for using PostgreSQL¶

To configure project you need to copy mysite/settings/local_conf_example.py into mysite/settings/local_conf.py and change DATABASES dict according to previously configured PostgreSQL user and db_name:

DATABASES = {
    'default': {
      'ENGINE': 'django.db.backends.postgresql_psycopg2',
      'NAME': 'db_name',
      'USER':  'your_username', (postgres for Ubuntu default configuration proccess)
      'PASSWORD': 'your_postgres_pass',
      'HOST': '',  # leave blank for localhost
      'PORT': '',  # leave blank for localhost
    }
}

Installing required Python Packages using pip¶

Assuming you have the above pre-requisites installed, within the socraticqs2 directory run the following command:

pip install -r dev_requirements.txt

Run the test suite¶

To make sure your setup is working properly, try running the test suite:

cd socraticqs2/mysite
python manage.py test

You should see a series of tests pass successfully.

By default test suite is running on sqlite database to get a speed boost but you can change this by commenting out next lines in you local_conf.py:

if 'test' in sys.argv or 'test_coverage' in sys.argv:
    DATABASES['default']['ENGINE'] = 'django.db.backends.sqlite3'

Warning

For Mac developer if you see ValueError: unknown locale: UTF-8 do the following:

export LC_ALL=en_US.UTF-8
export LANG=en_US.UTF-8

Or add this lines to your ~/.bash_profile

Running a test web server¶

You need to create a database, load it with some data, load plugin specifications, and then run the development web server.

You can prepare database with Fabric deployment tool.

To initialize DB to run test webserver use:

fab db.init

This task creates a new DB and transforms it into working state by the next steps:

Drop existing DB
Create new DB
Apply all current Django migrations
Load fixtures
Deploy all FSMs

To list all available tasks use:

fab --list

Finally, start up the development web server:

python manage.py runserver

Start a web browser and point it at http://localhost:8000/ct/

It will prompt you to log in; the default development tester username is admin with password testonly.

User admin is a conventional user to use during the FSM development.

You can stop the web server by simply typing Control-C.

Security notes:

you should not allow any outside access to localhost port 8000 as this is insecure (e.g. an outsider could log in with the default credentials above and modify data in your development server).

Basic Developer Operations¶

Getting source code updates, making changes etc.¶

See our Git Usage Guide.

Database Operations¶

Updating your database schema¶

If upstream code changes (i.e. made by someone else, and pulled into your local repo) alter the database schema, you will have to update your developer database to match. You will typically notice this in two ways:

upstream code changes introduced new migration files in ct/migrations/. These files tell Django how to update your database schema.
when you try to run the testsuite or runserver, you will get an error message saying that your database schema does not match the current data models.

To migrate your database to the new schema, type:

python manage.py migrate ct

Altering the database schema (models) yourself¶

If you change the database fields for a data model in models.py, you will of course also have to change your database to match. (Note that this means changes to the data fields that are stored in the database; changing or adding method code on the data classes does not change the database schema).

Django 1.7 makes this easy via its makemigrations command.

First make a backup copy of your current database (this is important, because it’s not obvious whether there is any easy way to “undo” a migration):

fab db.backup[:custom_branch_name]

It will create db backup file with suffix from you current git branch name or from provided optional param custom_branch_name:

fab db.backup:test1
.................
[localhost] local: pg_dump courselets -U postgres -w > /path/to/proj/backups/backup.engine.test1
[................

Done.

Then simply type:

python manage.py makemigrations ct

This will create a new migration file in ct/migrations. You then apply this migration to your database exactly as we did in the previous section:

python manage.py migrate ct

At this point you should be able to run the runserver, etc.

Warning

You must commit your new migration file at the same time as you commit your schema changes in models.py, so that others can update their database to match the new models. E.g. using command-line Git, you’d type something like:

git add ct/models.py
git add ct/migrations/0005_unitstatus.py
git commit -m 'added UnitStatus to models'

where ct/migrations/0005_unitstatus.py is the new migration file created by makemigrations to represent the changes you made to models.py.

Warning

There are several limitations that make migrations somewhat unwieldy. You need to be aware of the following “gotchas” lurking here:

once you change a model in models.py, your code will no longer run until you successfully run makemigrations + migrate. So you cannot actually move to manual testing your changes until you run both those steps.
every time you run makemigrations on another change to your data models, ANOTHER migration file will be added, and EVERY migration file will be required for the migration to work. Multiple migration files increase the risk of errors either in your committing them or other people attempting to apply them. So ideally, when you change the models to introduce a new feature, you want that to be represented by a single new migration file.
Because of this, in theory you shouldn’t run makemigrations / migrate until after you are pretty sure your model changes are final. But you can’t even start manual testing of your changes until after both steps. This is an unpleasant catch-22.
To run unittests you need to create migration according to your new code with no need to apply them to DB because test suite applies all migrations every time you run it to temporary DB.
Once you change your database schema (via migrate), all other code versions (i.e. not matching the new schema stored in your database) will NOT run. This would destroy the key virtue of Git – your ability to have many different code branches and switch between them effortlessly.
Because of this, to make developer life easier, we provide Fabriс db.init|db.backup|db.restore tasks to init, backup and restore actions with DB. See Fabric deployment tool section for details.

Recommended Migration Best Practices¶

For all these reasons, I suggest you follow a simple discipline whenever you are about to make model changes that will require migration:

BEFORE making those changes, make your DB backup and checkout a new Git branch, e.g.:

Make DB backup:
```
fab db.backup
```
By default without providing any custom params to task it will use you current branch name as suffix for backup file.

Next you can checkput to new bigchange:
```
git checkout -b bigchange
```
where bigchange is the name of your new branch.

You should also do this if you are starting to work with someone else’s experimental model changes, e.g.:
```
fab db.backup
git checkout -b bigchange
git pull fred bigchange
```
Then, if you ever want to switch back to your previous branch, you can simply switch back to the database file that worked with previous:
```
git checkout previous
```
Next you need to restore backuped DB:
```
fab db.restore
```
Note that you should NOT add any DB file to Git version control.
Now you can freely run makemigrations + migrate whenever you like, so you can test your changes.
If you DB is in initial state without any new information stored in you can use Fabric deployment tool to get DB ready to test and development:
```
$ fab db.init
```
This will give you a new DB with all migrations applied, initial data populated and all FSM deployed.
If it turns out that you need to make more model changes (i.e. your model changes turned out to be inadequate for the feature you’re implementing, and you haven’t yet committed the inadequate models/migration), the best practice is to UNDO your migration and REGENERATE a new migration to replace it, like this:
```
rm ct/migrations/0005_unitstatus.py
fab db.restore
python manage.py makemigrations ct
python manage.py migrate ct
```
where ct/migrations/0005_unitstatus.py is your new migration file.

If you migration can be backwarded you can use next flow:
```
python manage.py migrate ct 0004
rm ct/migrations/0005_unitstatus.py
python manage.py makemigrations ct
python manage.py migrate ct
```
where 0004 is a number of your previous migration file.

Backing up, flushing, and restoring your local database¶

You may wish to make and reload snapshots of your local database as part of your development and testing process. This is easy.

You can save a snapshot of your current database to a file, like this:

python manage.py dumpdata > dumpdata/mysnap.json

You can flush (delete all data) from your database like this:

python manage.py sqlflush|python manage.py dbshell

You can then restore a particular snapshot like this:

python manage.py loaddata dumpdata/mysnap.json

Test project on PostgreSQL before making any PR¶

You can develop project using sqlite db but before creating any PR you need to init PostgreSQL DB to ensure your code not breaking anything.

You can create and populate db with Fabric deployment tool:
```
fab db.init
```

This will create and populate db with fixture. Then deploy all FSMs.

Run all tests:
```
python manage.py test
```
Run test server and start Course from home page.
Go to Activity Page and restore Activity.
Go to Activity Page and cancel Activity.

Getting Started with Socraticqs2 Development¶

Installation and setup¶

Recommended: use virtualenv to create an isolated test environment¶

Pre-requisites¶

Installing Git version control software¶

Getting the Socraticqs2 source code¶

Installing and configuring PostgreSQL¶

Ubuntu 14.04 distribution instructions¶

Mac OS X installation¶

Configuring Django project for using PostgreSQL¶

Installing required Python Packages using pip¶

Run the test suite¶

Running a test web server¶

Basic Developer Operations¶

Getting source code updates, making changes etc.¶

Database Operations¶

Updating your database schema¶

Altering the database schema (models) yourself¶

Recommended Migration Best Practices¶

Backing up, flushing, and restoring your local database¶

Test project on PostgreSQL before making any PR¶

Table Of Contents

Previous topic

Next topic

This Page