Wednesday, November 6, 2013

When will Sagemath Cloud hit 100,000 user accounts?

The Sagemath Cloud is an online environment for computational mathematics. Get an account, log in, and your web-browser transforms into almost everything you need to study algebra, calculus, numerics, statistics, number and game theory, (computational aspects of) physics, chemistry and all other quantitative sciences. It's built around all sorts of tools and utilities offered in a proper Linux environment. All files can be shared with collaborators and edited in real-time without stepping on each other's toes. Oh wait, there is also a LaTeX editor: given you know LaTeX, qualitative sciences are also covered ;-)

But beware, this posting will only talk about the very first: accounts. Below, I'll explain how I managed to create this page, the plot below and how I let it update automatically every hour - all done solely by SMC!

 SMC accounts over time

Sagemath Cloud is very public about what is happening on its servers. This stats link gives you the raw data for the current overall load of the machines. What I did over the last weeks was the following: Every hour I've downloaded this stats json file, parsed it, accumulated some interesting numbers, and stored the processed data in a CSV file.

I did all this in SMC, because it allows you to run your own crontab files. Cron periodically goes through all kind of files to figure out if there is a job to do. Just enter a magical line in your own crontab file, and the given command is run whenever you tell it to do so. You do not have to be logged in!

In my case, it's like this:
$crontab -e Then enter this line: 0 * * * * python$HOME/get.py

The \$HOME is important, because you have to specify the full path where your script sits. (By the way, if you have to start something when the SMC server reboots, use the @reboot descriptor)

So, what is get.py doing? It uses the wonderful requests Python library to retrieve and parse the stats url and extracts some data. Then it appends a line to a CSV file.

Two minutes later (the crontab line starts with a "2"), another script is called which processes this CSV file. It reads the columns via pandas, properly parses the dates into time-series, and allows me to do all sorts of analysis, transformations and plots with it. For example, the plot shown above overlays the raw time series plot with OLS fits done by statsmodels for selected time ranges (where it looks "flat"). We can see the growth trends clearly! Even more important, so far the growth increases and hence we are watching the exponential growth phase as part of the beginning of the usual logistic growth.

The other plots on this statistics page show aggregated time-series data. For example, the plot for the number of concurrent connections is also increasing and reassures that SMC is scaling well.
 Concurrent connections to SMC
On this statistics page, there are also a few "dynamic" fields in the HTML content. This is done by jinja2 in such a way, that the template "stats.tmpl" contains the HTML code and "mustache"-style variables. Jinja2 renders this template with some variables and that's it.

import jinja2 as j2
stats = env.get_template("stats.tmpl")
data = {
'date' : "%s UTC" % datetime.utcnow(),
'recent_data' : totals.ix[-24:].to_html()
}
with open("stats.html", "wb") as output:
output.write(stats.render(**data))


The last step of the script is to actually publish the files to the webserver. That's rather straight forward. First and only once, create ssh keys via ssh-keygen without a password and then use ssh-copy-id -i ~/.ssh/id_dsa.pub name@server to copy over your keys. Subsequent ssh connections will be established without any questions asked, because the remote server knows your identity. I'm using scp to copy the files: scp *.png stats.html name@remote-server:~/target/dir/

Last but not least, when will SMC hit the 100,000 user mark? In less than 6 days the 10,000 mark should be crossed and I hope the trend continues into the upward direction.

Monday, May 27, 2013

Sage announces 3 GSoC projects

Sage is pleased to announce three Google Summer of Code projects for 2013. They focus on speed improvements of symbolic functions, simplifying the distribution and installation procedure on Debian/Linux and ubiquitous accessibility of Sage on the Android platform.

Mathematical Functions Library

Eviatar Bach –  University of British Columbia in Vancouver, Canada
(Mentor: Flavia Stan, Backup: Burcin Erocal)

Sage interfaces with multiple third-party libraries, such as MPFR, GSL, GP/PARI, mpmath, and Maxima, for numerical evaluation of special functions. There are significant discrepancies between these backends in the performance for numerical approximations of the same expression. An initial benchmark reveals, for example, that calculating spherical_bessel_J(1, 5.2) with SciPy is over 100 times faster than with Maxima.

The project has the following goals:
1. develop a benchmark framework to determine which backend should be used by default to evaluate a special function over a specific domain,
2. create symbolic wrappers for all the special functions that can be evaluated numerically by a package included in Sage,
3. create a data structure for generalized hypergeometric functions and extend the symbolic wrappers to obtain representations in terms of generalized hypergeometric functions when possible,
4. implement closure properties for holonomic functions as a next step to improve the symbolic processing of special functions in Sage.

Overall improvement of the Sage Android application

Rasmi Elasmar
(Mentor: Volker Braun, Backup: Harald Schilly)

Although there are already some existing efforts, Sage is still not easily accessible from the Android platform. The Sage Cell client/server infrastructure is an already existing step towards running Sage on a server and communicating back the results. The aim of this proposal is to fix, improve and update the Sage Android application to include new features and functionality, as well as an improved interface for simpler and improved usability. Android's new "Holo" style, sharing of calculations and results, and much more waits to be realized on Android for Sage.

Get Sage ready for Linux distributions

Felix Salfelder – Goethe Universität, Frankfurt, Germany
(Mentor: Tobias Hansen, Julien Puydt, Jeroen Demeyer & John Palmieri )

The aim of this project is to detach the build process of Sage ("the software") from Sage ("the distribution"). The goal is a build system that works within the context of Sage as well as for any GNU/Linux distribution that ships the dependencies for Sage. Distributions that already ship Sage packages or plan to do so are Fedora and Debian. This project is an important step towards making Sage packages in GNU/Linux distributions feasible.

Sage warmly welcomes all three new students and wishes them all the best to learn something new and make an impact in Sage's future developments!

Monday, April 8, 2013

Sage part of Google's Summer of Code 2013

Good News, like last year Sage is once again part of Google's Summer of Code. This means, until April 22 at 19:00 UTC students can submit their applications and mentors will review them and do the matching.