Cant rename/remove /var/tmp in 12.3?

[Josef Wolf]

Hello,

in all my linux installations, I used to replace /var/tmp by a symlink to a
directory on my main user partition. The intent of this is to ensure that
there is no private information on the root partition.

With 12.3, this won't work anymore:

raven:/ # mv /var/tmp /var/tmp-orig
mv: cannot move ‘/var/tmp’ to ‘/var/tmp-orig’: Device or resource busy

raven:/ # ls -ld /var/tmp
drwxrwxrwt 15 root root 4096 Jul 31 10:41 /var/tmp

raven:/ # getfacl /var/tmp
getfacl: Removing leading '/' from absolute path names
# file: var/tmp
# owner: root
# group: root
# flags: --t
user::rwx
group::rwx
other::rwx

raven:/ # df /var/tmp
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/cr_sda8 20649592 8361332 11239320 43% /

I get the same error when trying to delete. Even rebooting into single user
mode won't help.

Any ideas why the directory appears to be busy? Even if I remove all the
contents of the directory, it still appears to be busy. BTW: a busy directory
seems to be totally new semantics to me.

[Bernhard Voelker]

Post by Josef Wolf
Hello,
in all my linux installations, I used to replace /var/tmp by a symlink to a
directory on my main user partition. The intent of this is to ensure that
there is no private information on the root partition.
raven:/ # mv /var/tmp /var/tmp-orig
mv: cannot move ‘/var/tmp’ to ‘/var/tmp-orig’: Device or resource busy
raven:/ # ls -ld /var/tmp
drwxrwxrwt 15 root root 4096 Jul 31 10:41 /var/tmp
raven:/ # getfacl /var/tmp
getfacl: Removing leading '/' from absolute path names
# file: var/tmp
# owner: root
# group: root
# flags: --t
user::rwx
group::rwx
other::rwx
raven:/ # df /var/tmp
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/cr_sda8 20649592 8361332 11239320 43% /
I get the same error when trying to delete. Even rebooting into single user
mode won't help.
Any ideas why the directory appears to be busy? Even if I remove all the
contents of the directory, it still appears to be busy. BTW: a busy directory
seems to be totally new semantics to me.

man 2 rename:

EBUSY The rename fails because oldpath or newpath is a directory that
is in use by some process (perhaps as current working directory,
or as root directory, or because it was open for reading) or is
in use by the system (for example as mount point), while the
system considers this an error. (Note that there is no require-
ment to return EBUSY in such cases—there is nothing wrong with
doing the rename anyway—but it is allowed to return EBUSY if the
system cannot otherwise handle such situations.)


That means the kernel could refuse to rename(2) the directory if it was
the current directory - but the openSUSE kernels permit it.

Another reason would be an active mount point:

linux-gbg1:~ # mkdir d

linux-gbg1:~ # mount -t tmpfs tmpfs d

linux-gbg1:~ # strace -e rename,stat,lstat mv d d-orig
stat("d-orig", 0x7fff2a9143c0) = -1 ENOENT (No such file or directory)
lstat("d", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=40, ...}) = 0
lstat("d-orig", 0x7fff2a914080) = -1 ENOENT (No such file or directory)
rename("d", "d-orig") = -1 EBUSY (Device or resource busy)
mv: cannot move ‘d’ to ‘d-orig’: Device or resource busy
+++ exited with 1 +++

Your commands above show that /var/tmp is not the mount point (but "/" is").
How does your strace look like?

In a plain installation, I can't imagine much other reasons for EBUSY.
Did you play with some other restricting methods like SELinux etc.?
Are there suspicious entries in /proc/self/mounts?
Does "lsof /var/tmp" or "fuser /var/tmp" show something, e.g. a process
chroot-ed to /var/tmp?

Have a nice day,
Berny

[Josef Wolf]

Thanks for the quick response, Bernhard!

Post by Bernhard Voelker
Your commands above show that /var/tmp is not the mount point (but "/" is").
How does your strace look like?

raven:/ # strace -e rename,stat,lstat mv /var/tmp /var/tmp-orig
rename("/var/tmp", "/var/tmp-orig") = -1 EBUSY (Device or resource busy)
mv: cannot move ‘/var/tmp’ to ‘/var/tmp-orig’: Device or resource busy
+++ exited with 1 +++
raven:/ #

Post by Bernhard Voelker
In a plain installation, I can't imagine much other reasons for EBUSY.
Did you play with some other restricting methods like SELinux etc.?

No. Everything is as it was on a fresh install.

Post by Bernhard Voelker
Are there suspicious entries in /proc/self/mounts?

raven:/ # grep tmp /proc/self/mounts
devtmpfs /dev devtmpfs rw,relatime,size=1530924k,nr_inodes=207269,mode=755 0 0
tmpfs /dev/shm tmpfs rw,relatime 0 0
tmpfs /run tmpfs rw,nosuid,nodev,relatime,mode=755 0 0
tmpfs /sys/fs/cgroup tmpfs rw,nosuid,nodev,noexec,mode=755 0 0
tmpfs /tmp tmpfs rw,nosuid,nodev,noexec,relatime,size=524288k 0 0
tmpfs /var/lock tmpfs rw,nosuid,nodev,relatime,mode=755 0 0
tmpfs /var/run tmpfs rw,nosuid,nodev,relatime,mode=755 0 0
raven:/ #

Post by Bernhard Voelker
Does "lsof /var/tmp" or "fuser /var/tmp" show something, e.g. a process
chroot-ed to /var/tmp?

Nothing chrooted:

raven:/ # ls -ld /proc/*/root|grep -v ' /$'
lrwxrwxrwx 1 root root 0 Jul 31 12:09 /proc/3309/root -> /var/lib/ntp
lrwxrwxrwx 1 root root 0 Jul 31 12:11 /proc/3892/root -> /proc

lsof and fuser show a lot of entries, but none of them are related directly to
/var/tmp.

In the 20 years I have been using linux, I have never seen that changing into
a directory or keeping files open (even for writing) in a direcoty would
prevent you from deleting/renaming that directory. Not even chroot would
prevent renaming (I just tried with /var/lib/ntp shown above).

While deletion could fail if deletion of some file or directory further down
fails, I don't see any reason why a rename could possibly fail.

And I strongly consider those semantics to be a feauture, not a bug. Just take
a look over there to the windows world, where all applications need to be
stopped every time you want to install or update something.

The only reason I can see would be a mount. But there is nothing mounted on
/var/tmp.

[Bernhard Voelker]

Post by Josef Wolf
The only reason I can see would be a mount. But there is nothing mounted on
/var/tmp.

Agreed.

Last idea: the kernel returns EBUSY if it holds an internal reference to
the directory. Is that a strange or special file system?
I.e. something like BTRFS and 'var' being the name of a snapshot?
(I've not played much with BTRFS yet).

Have a nice day,
Berny

[Josef Wolf]

Post by Bernhard Voelker

Post by Josef Wolf
The only reason I can see would be a mount. But there is nothing mounted on
/var/tmp.

Agreed.
Last idea: the kernel returns EBUSY if it holds an internal reference to
the directory. Is that a strange or special file system?
I.e. something like BTRFS and 'var' being the name of a snapshot?
(I've not played much with BTRFS yet).

It's a standard ext3 as the suse-installer defaults to it:

raven:/ # cat /proc/mounts | grep " / "
rootfs / rootfs rw 0 0
/dev/mapper/cr_sda8 / ext3 rw,relatime,data=ordered 0 0
raven:/ #

Although playing with snapshots for backups (on ext4) is on my todo-list, I've
not been on that route yet.

I have installed 12.3 exactly the same way as I had installed the countless
versions before it. I have a hand-written script of instructions that I follow
when installing and I only derivate from it when changes in the opensuse
installer force me to do so.

After installation the machine is configured by a system similar to
cfengine. In fact, this configuration system stumbled over this /var/tmp
problem.

So I am pretty sure that I have _not_ installed or configured anything
different than in previous versions.

How can I check whether SELinux might be a problem here? I've never been into
SELinux yet...

What about policykit? Isn't that something similar to SELinux?

Please, can _you_ try to rename /var/tmp?

[Bernhard Voelker]

Post by Josef Wolf
Please, can _you_ try to rename /var/tmp?

It works here:

$ strace -e rename mv /var/tmp /var/tmp-orig
rename("/var/tmp", "/var/tmp-orig") = 0
+++ exited with 0 +++

Hmm, your data is encrypted, isn't it?

Post by Josef Wolf
/dev/mapper/cr_sda8 / ext3 rw,relatime,data=ordered 0 0

I've never used it, but possibly there's something wrong on
that level or the underlying device.
Did you do an fsck on "/"?

And a silly question: you are not in a virtualized system,
are you?

Have a nice day,
Berny

[Josef Wolf]

Post by Bernhard Voelker

Post by Josef Wolf
Please, can _you_ try to rename /var/tmp?

$ strace -e rename mv /var/tmp /var/tmp-orig
rename("/var/tmp", "/var/tmp-orig") = 0
+++ exited with 0 +++

Strange

Post by Bernhard Voelker
Hmm, your data is encrypted, isn't it?

Post by Josef Wolf
/dev/mapper/cr_sda8 / ext3 rw,relatime,data=ordered 0 0

I've never used it, but possibly there's something wrong on
that level or the underlying device.

Yes. I've been using LUKS for many years with different distros. Never had
any problems.

Post by Bernhard Voelker
Did you do an fsck on "/"?

Not yet. Can't shut down the system right now.

Post by Bernhard Voelker
And a silly question: you are not in a virtualized system,
are you?

No. I always use real systems.