• 9 Posts
  • 87 Comments
Joined 3 years ago
Cake day: August 10th, 2023
  • moonpiedumplings@programming.devtoLinux@lemmy.ml•Is multi user nix safe?English
    3·
    1 month ago

    Yes, although there have been a few CVE’s related to escalating privileges or breaking out of the sandbox. You’re going to want to keep those updating and keep on top of those.

    There is one concern, in that the nix store (/nix/store) is world readable. It is not world writable, which is good, but there is a problem in that secrets can potentially be copied into the nix store. If you copy a file containing environment variables or the like into the nix store, it could theoretically be found it. This one is on the users of nix to be careful of.

    With Nix flakes, the entire git repo that the flake originates from is copied into the nix store. Meaning if you have a nix flake in something that is supposed to be a private repo, or contains tracked sensitive data (untracked files are not copied into the nix store), then it could be found and inspected by other users.

    My big concern with multi user in this case is not merely the Nix daemon though, but also Linux itself. Linux has been hit with a LOT of privilege escalation and container escape issues over the past few years, and many of them have been zero day’s. Given this, I no longer really have the same level of trust for Linux with regards to multi user isolation, for in cases like these.

    Of course, in academic computing, I would probably just do multi user anyways, for simplicity. If you install tracking and monitoring, then you can attach malicious actions to identities. Because every user is registered and operates within the institution, if they break the terms of use for computing equipment, then you can punish them within the institution, or pursue greater legal action.

    And then, you also would want to enforce 2FA to help minimize stolen accounts. While that can still be phished via fake login pages that ask for 2FA, it handles most of the things. Isolate the server via firewalls, and then it becomes a small enough target that doesn’t have enough value (assuming of course, your research isn’t too valuable and worth targeting), and you mostly don’t have to worry about it.

    Sure, people will root it. But then they’ll send you a nicely worded email explaining how they rooted it and how to fix it. Which is what my friend did at my school, on our shared ssh server. Just keep it updated to handle the low hanging fruit.

    However, if you are going to give it to untrusted users with unknown or temporary (not within the institution) identities, then things change, and you have to take it a lot more seriously. I no longer have confidence in just Linux’s user based isolation.

    The first line solution I would go to, is to put users in containers and mount the nix store (and nix daemon) in and out. Something like a docker/podman container or Incus container. Of course, container escapes are still possible. If you are even more concerned about those, then you would want virtual machines. It is still possible to share the nix store between virtual machines, but it is more complicated.

    In addition to that, virtual machines have a performance tradeoff (usually 95% or more of the host’s performance though), but there might be issues with sharing GPU compute resources among virtual machines, depending on the hardware you are using. Enterprise GPU’s usually support it though.

  • moonpiedumplings@programming.devtoLinux@lemmy.ml•HPC system setupEnglish
    1·
    2 months ago

    Containers are commonly used to distribute programs that depend on different library versions, including different libc versions.

    And yes, you could compile software for specific hardware and then deploy it via containers.

    What is the advantage in using a container to say run gromacs, rather than pointing the user to the path of the compiled binary

    Different libc versions. There are also some sandboxing and security restrictions that are applied.

    I definitely want to have some persistent services running: databases, a couple web applications and maybe a Jupiter webpage to visualize results.

    Maybe docker swarm mode, with the nvidia container runtime is easier? It lets you spin up services on multiple nodes, which you could then load balance with a conventional reverse proxy. Although if you are trying to do a single instance of Jupyter, then I don’t know about it’s ability to serve multiple users, or the security of that setup. Usually, people go for things like JupyterHub + the Dockerspawner. Jupyterhub handles authentication or

    There is also a Kubernetes spawner, or you can use an alternative web application that can dynamically spin up jupyterlab/notebooks, like Kubeflow, or coder, but I don’t think any of that is what you want in regards to that specific usecase for Jupyter.

    If you just want to present rendered stuff, check out: https://quarto.org/ , which is a static site generator capable of executing jupyter notebooks, and rendering them to websites. It can also render I use it for my blog, which is ironic because I don’t actually use any of the data science features quarto has. I really like quarto because it’s the only static site generator which I found has fulltext search built in/easily enabled. It runs some javascript over a generated index, and you can search the website without any form of backend needed.

  • moonpiedumplings@programming.devOPtoLinux@lemmy.ml•[Solved] Benefit of a subvolume below the top level btrfs subvolume?English
    2·
    2 months ago

    Two years later, and I have an answer, after reading this: https://archive.kernel.org/oldwiki/btrfs.wiki.kernel.org/index.php/SysadminGuide.html#Subvolumes

    which was linked from the Arch Wiki, but it is the old wiki, which is obsolete and no longer updated. Basically, in a nested subvolume layout, the nested subvolumes inherit mounting options of their parent subvolume. This might be changeable, but it’s the default.

    On the other hand, separate subvolumes have their own mount options. Although not really important for a swap file, since the file itself can have copy on write features removed, you might want this if you want a subvolume containing folders of data where you want copy on write disabled for performance purposes. This matters for things like postgres, which has it’s own alternatives to journaling/CoW:

    Because WAL restores database file contents after a crash, journaled file systems are not necessary for reliable storage of the data files or WAL files. In fact, journaling overhead can reduce performance, especially if journaling causes file system data to be flushed to disk. Fortunately, data flushing during journaling can often be disabled with a file system mount option, e.g., data=writeback on a Linux ext3 file system. Journaled file systems do improve boot speed after a crash.

    From here: https://www.postgresql.org/docs/18/wal-intro.html

    And then the other benefit of a top level subvolume is management. You can mount it anywhere you want. You can mount it independently of other btrfs subvolumes, meaning you could share a swap file between two installed distros to save space, although this breaks hibernation, so you probably wouldn’t want that. It’s just that top level subvolumes give you flexibility nested subvolumes don’t.

  • moonpiedumplings@programming.devtoLinux@lemmy.ml•HPC system setupEnglish
    1·
    2 months ago

    Firstly, you should check out what the organization you are building for uses. If they use Red Hat, or Ubuntu, then you should probably just build your solution on top of those operating systems (or a Red Hat clone like Rocky or Alma). Both of those are popular in many organizations, and it would probably be better to use what people are familiar with and know how to troubleshoot or work their way around. Potentially, they even have support contracts, giving them the option of calling the parent company for help.

    Ansible is good. The learning curve is definitely less steep than Nixos, and it’s easier to teach people. One benefit is that you can reuse existing public roles and playbooks. For example this one: https://github.com/galaxyproject/ansible-slurm , which installs slurm.

    You would probably have to write additional playbooks or roles to install nvidia drivers or configure the system, but then they can stick.

    openhpc

    Firstly, is there anything specific from here you need? Secondly, is there anything in here that’s not available in existing distro repositories, like Ubuntu or Red Hat’s?

    It certainly looks like an interesting project, but a 3 node cluster is pretty small, and I find it hard to justify things like OpenMP/MPI, which is basically a special compiler that compiles programs to run across multiple machines at once. For that runtime to work, you do actually have to compile the programs, which are written for it, using it, which can require work on the side of the people who want to run applications or simulations.

    The more likely setup, to me, is that Slurm is going to to run docker containers via Apptainer. Slurm would handle assigning containers to nodes based on free resources, but they wouldn’t actually share resources like memory or CPU. This setup is still plenty useful, and very common.

    Original comment, from before I read that you already selected slurm below. This comment isn’t relevant, as I realized I was targeting the wrong things but I’ll just leave it here regardless.

    For the platform itself, you should use either Kubernetes or Slurm. Slurm is popular in academia, and Kubernetes is popular in corporate, but they are used interchangably depending on specific needs.

    Slurm is better for scheduled tasks. Like let’s say you want researchers at a school to be able to run a long running simulation. They can sign up, reserve time for that slurm cluster, and then send out a “job”, for that slurm cluster. The job will automatically be allocated to the node of the cluster with free resources, and then ran, and then stopped, and the researcher will receive the results back.

    Kubernetes is better for persistent deployed services, like web services or the like. For example, AI inference. Kubernetes can also do batch jobs, but it doesn’t have the advanced time tracking or scheduling systems that slurm has (although sometimes people build that on top of Kubernetes in order to only deploy one clustered system).

    Nixos is nice for configuration as code, but it lacks clustering or application/cluster orchestration features. You can use Nixos to deploy Kubernetes or Slurm, but I wouldn’t use it as the HPC platform itself. Nixos should work with Cuda in addition to that.

  • moonpiedumplings@programming.devtoLinux@lemmy.ml•Is it worth attempting to make a libre version of QubesOS?English
    10·
    3 months ago

    Absolutely, this is an admirable project.

    But the first thing you should know is that QubeOS is not based on Linux, but is actually a different kernel, Xen. The Xen kernel virtualizes Linux, and all Linux runs under it.

    Networking and hardware access is done by certain VM’s having devices (like the ethernet card or monitor/keyboard) passed through them, where Linux then handles the hardware access with it’s drivers.

    This is important to understand that, because QubeOS is not a Linux distro. Really, it’s just that they selected Linux (it was either Debian or Fedora IIRC) as their management VMs.

    But once you understand that it’s absolutely feasible to adjust the VM’s. It’s probably easier to modify a LInux distro (or create your own) and use that for all the VM’s, and then to reuse Qube’s management related software. That way you could do something like ship a version of debian that disables non-free software and firmware in the debian repos. Doing that is probably easier than creating your own distro entirely from scratch.

    Another interesting thing about Qubes is that you are not limited to Linux. Of course, using Linux will be easiest. But the management VM’s can technically be any OS that supports it.

  • moonpiedumplings@programming.devtoLinux@lemmy.ml•What reasons do people have for disliking SELinux?English
    3·
    4 months ago

    Kind of.

    Copyfail would punch through user namespaces to get root straight on the host. User namespaces only really protect you against vulnerabilities in non kernel applications.

    Limited capibilities/seccomp policies did help, though. In my admittedly limited testing, some of the vulnerabilities wouldn’t work in podman, but they would work in docker. This wasn’t due to user namespaces, but this was due to podman having stricter capibilities/seccomp policies than docker by default.

    This implies that even if you were using docker rootless, they still would have been able to break out and get root in one go.

    User namespaces don’t add that much security, in my opinion. Assuming your container has a non root user inside, adding user namespaces just changes the amount of cve’s/zerodays from 2 to maybe 3:

    With a rootful container it’s:

    • Escalate to root (can be done after or before container escape)
    • Escape container (can be done after or before escalation to root)

    With user namespaces it becomes:

    • Maybe escalate to root within the container first to get privileges or access to binaries needed to take advantage of a container escape exploit
    • Escape container
    • Escalate to root

    User namespaces are like every other Linux security solution, they are extremely complex, hard to configure, and they don’t actually add that much security for the trouble The article I linked above has a section about them:

    Another example of these features is user namespaces. User namespaces allow unprivileged users to interact with lots of kernel code that is normally reserved for the root user. It adds a massive amount of networking, mount, etc. functionality as new attack surface. It has also been the cause of numerous privilege escalation vulnerabilities, which is why many distributions, such as Debian, had started to restrict access to this functionality by default

    Their complexity makes them difficult to secure and execute properly, and adds a ton of attack surface to the kernel.

    Dirty frag, for example, was using user namespaces as one of the ways it would escalate. Most container runtimes restrict user namespace creation within user namespaced containers (via seccomp/capabilities), so running dirtyfrag in a container wouldn’t have worked. But, at the same time, dirtyfrag is only possible in the first place because of the attack surface user namespaces cause.

    I mostly use docker and rootfull podman for everything. You already need a CVE/zeroday to do a container break out in the first place, so just keep your runtimes up to date and you should be good. If you really care about being proactive with security, and trying to preemptively prevent issues, user namespaces are not really a good solution, better is just to use a VM container runtime like kata or microvm, or a userspace kernel like gvisor or syd. They are pretty easy to use. You can just set them as your container runtime, in docker, podman, or kubes, and things will mostly just work. Those (and other kernel isolation solutions) would have actually beaten dirtyfrag, copyfail, and the like of recent vulns.

  • moonpiedumplings@programming.devtoLinux@lemmy.ml•What reasons do people have for disliking SELinux?English
    17·
    4 months ago
    1. It’s extraordinarily complex.

    The reality is that security is not just technical implementation, but also actually getting people to use the solutions. “Stop disabling SELinux” is not a real answer to when people disable it, like we have one person in this thread.

    Another problem with complex security solutions is they are hard to get right. Even if you enable them and configure them, without being an expert, it’s possible you left a gap here or there, and holes and gaps in these solutions.*

    1. Like so many other complex linux security solutions, it is lacking effectiveness due to still sharing the same kernel.

    There is a good, but bit dated writeup here about the problems with Linux security, from an architecturual perspective: https://madaidans-insecurities.github.io/linux.html . But, the short version is that the Linux kernel is large and complex, and has a lot of attack surface. And it’s a frequent source of vulnerabilities because attackers can hit it as long as they access to the kernel, even if they are in a container/sandbox. Like, copyfail and dirtyfrag would punch through containers, but also punch through SELinux.

    For example, just earlier on lemmy someone dropped a zero day that punches through SELinux: https://programming.dev/post/51103657

    Now, SELinux can be used to restrict what a root shell could do after escalating… but that’s further complexity you have to learn to configure, and configure it correctly as well.

    Ultimately, none of the Linux security solutions come anywhere near the isolation of simply running something in a virtual machine. Which, also happens to be a lot simpler and actually possible to get people to use.

    *(putting this at the bottom because it veers off topic) I have a greater argument and problem with mentalities like this. I have noticed a pattern, where many of the more effortfull and toil intensive security solutions are recommended by people who have the time, energy, and skills to execute them. They have a bias/blindspot to the realities, which is that not everyone is in the same situation as them.

    For example, updating/patching software. Linux distros like RHEL or Debian, have a policy where they only do security updates, and don’t do feature updates or bugfixes. This enables them to ship automatic updates, so that security issues are automatically handled.

    On the other hand software like Windows, likes to bundle in breaking changes along with security updates. So automatic updates get disabled because “They might break something”. And then, people don’t update them, and environments get horrifically out of date, because not enough money/time/people is put into regular IT people who are in charge of maintaining them.

    But some environments, have heroes, people who go around patching everything and keeping everything up to date and secure. And when they see these environments that don’t have everything patched, they usually give the advice of “You should patch everything” (while simultaneously advising against auto updates), not understanding that these environments are lacking a key ingredient: Themselves.

    Sure, I could be a hero. I could “patch” everything manually. I could deploy SELinux. But that would only last until I get burnt out, or leave. Once I’m gone, SELinux, the patches, any similar security solutions are gone. I’ve met so many people, even in cybersecurity, that are apathetic about security, even though they might have cared once upon a time.