Pkg: Julia's Built-in Package ManagerFirst Posted: July 30, 2022
Last Updated: August 27, 2022
In this page, we will quickly and easily create a new project environment in Julia using its own package manger, named Pkg.
Since we're pretty early on in these tutorials at this point, I will hold off on a lot of detail as to why making a local project environment is preferred over maintaining a large global environment. But I'll give two reasons for now just to keep your interest:
Smaller local projects typically have fewer dependencies, and therefore updating those dependencies is faster and less prone to conflicts.
By leveraging Pkg, we can make our work extremely portable across different machines. This means that I can send someone else my code and just a couple of small text files and boom! they can recreate everything I had and completely reproduce my work.
These two reasons actually make collaborating with Julia projects very simple and great for scientists. How many times have you tried downloading some code and it can't work out of the box? This method of local environments really helps reduce this risk by improving the portability and reproducibility of scientific projects written in Julia.
PkgTo access the Pkg from the Julia REPL, simply hit ] and see that the julia> heading change to pkg>. Now, to create a new project environment, type activate . like so:
(v1.8) pkg> activate .This will change the REPL heading again from pkg> to (fish) pkg>, given a directory name fish. So now we should be looking at
(fish) pkg>If you hit Backspace, you leave Pkg mode and will see julia> again. In an empty fish directory, typing
julia> readdir()should yield nothing. This is because our newly activated fish project has no dependencies. So let's add a couple. Hit ] to enter Pkg mode and type
(fish) pkg> add Statistics StaticArraysand hit Enter. This will download the supplied list of packages from Julia's General Registry by default. In this case we asked for two packages. When it's done, hit Backspace to leave Pkg mode, and then type
julia> readdir()again. Now, you should see two new files: Project.toml and Manifest.toml.
Congrats! You've now set up a new Julia project environment called fish that has two dependencies: Statistics.jl and StaticArrays.jl. At this point, you could continue to play around within this environment, add more packages, delete others, etc., without having your actions locally affect your global Julia environment[1].
TOML filesHere I'll briefly discuss what these TOML files are and why Julia generated two of them.
Project.toml fileThis file is used to uniquely specify a new project environment and list all of the packages it depends on – its dependencies. If this file already exists in a directory when one goes to activate . a new environment, the present Project.toml file will be used instead of creating a new one. So in this sense the local project environment is re-activated.
Manifest.toml fileThis file uniquely delineates not only the your new project environment's dependencies, but also all of the dependencies' dependencies, and so on. The power of this file comes from another Pkg command:
(v1.8) pkg> instantiate .which will not only pull the dependencies listed in the Project.toml, but also all of the specific package versions of every package listed in the Manifest.toml. This file and instantiate command is what allows for portability and reproducibility of Julia projects because it allows anybody to completely recreate the specific environment configuration that you used to run your code and get your specific results.
Pretty cool, right?
TOML file?If a lot of the power comes from the Manifest.toml, why not just use this one?
It turns out that because of the way the Julia team have designed their package managment system, people who actually write Julia packages rarely need to include their Manifest.toml. Indeed, actually when one writes a package, they typically want it to be used by as many people as possible, without limiting their use cases by constraining them to a specific configured environment. So they test their packages on servers without sending the Manifest.toml[2], and let the test servers create their own local environments. If everything works out, then they can add their package to the General Registry, and we can download and use it in our own environments.
Hooray for code reuse!
It is only when we need, say a collaborator, advisor, or TA, to reproduce our work that we absolutely need to send the Manifest.toml. In that case, they would want to be constrained by our specific local environment to obtain the same results we did.
Hooray for reproducibility in science!
At this point, we've covered the absolute basics as far as using Julia's Pkg manager to create local project environments.
We created the fish environment with two dependencies, Statistics.jl and StaticArrays.jl, both of which we pulled from Julia's General Registry of packages. By adding these dependencies, we saw that Pkg.jl automatically generated a Project.toml and Manifest.toml in our project directory. We then discussed the differences between these files, and how the Julia team made it so that just these files make Julia code reusable, portable, and reproducible.
I've tip-toed around some other important details to avoid overwhelming you with so much information, especially if you're new, not just to Julia, but programming in general. For anyone interested in more information, you can find it at:
Clicking on the specific footnote number will take you back to where you were in the text.
| [1] | The global Julia environment is named v1.8 for Julia versions 1.8.x. To access it again from the fish environment, or any other, simply write
(emphasis on no |
| [2] | Package authors do have to send the Project.toml though.
|