Freezing of tasks	(C) 2007 Rafael J. Wysocki <[e-mail]>, GPLI. What is the freezing of tasks?The freezing of tasks is a mechanism by which user space processes and somekernel threads are controlled during hibernation or system-wide suspend (on somearchitectures).II. How does it work?There are four per-task flags used for that, PF_NOFREEZE, PF_FROZEN, TIF_FREEZEand PF_FREEZER_SKIP (the last one is auxiliary).  The tasks that havePF_NOFREEZE unset (all user space processes and some kernel threads) areregarded as 'freezable' and treated in a special way before the system enters asuspend state as well as before a hibernation image is created (in what followswe only consider hibernation, but the description also applies to suspend).Namely, as the first step of the hibernation procedure the functionfreeze_processes() (defined in kernel/power/process.c) is called.  It executestry_to_freeze_tasks() that sets TIF_FREEZE for all of the freezable tasks andeither wakes them up, if they are kernel threads, or sends fake signals to them,if they are user space processes.  A task that has TIF_FREEZE set, should reactto it by calling the function called refrigerator() (defined inkernel/power/process.c), which sets the task's PF_FROZEN flag, changes its stateto TASK_UNINTERRUPTIBLE and makes it loop until PF_FROZEN is cleared for it.Then, we say that the task is 'frozen' and therefore the set of functionshandling this mechanism is referred to as 'the freezer' (these functions aredefined in kernel/power/process.c and include/linux/freezer.h).  User spaceprocesses are generally frozen before kernel threads.It is not recommended to call refrigerator() directly.  Instead, it isrecommended to use the try_to_freeze() function (defined ininclude/linux/freezer.h), that checks the task's TIF_FREEZE flag and makes thetask enter refrigerator() if the flag is set.For user space processes try_to_freeze() is called automatically from thesignal-handling code, but the freezable kernel threads need to call itexplicitly in suitable places or use the wait_event_freezable() orwait_event_freezable_timeout() macros (defined in include/linux/freezer.h)that combine interruptible sleep with checking if TIF_FREEZE is set and callingtry_to_freeze().  The main loop of a freezable kernel thread may look like thefollowing one:	set_freezable();	do {		hub_events();		wait_event_freezable(khubd_wait,				!list_empty(&hub_event_list) ||				kthread_should_stop());	} while (!kthread_should_stop() || !list_empty(&hub_event_list));(from drivers/usb/core/hub.c::hub_thread()).If a freezable kernel thread fails to call try_to_freeze() after the freezer hasset TIF_FREEZE for it, the freezing of tasks will fail and the entirehibernation operation will be cancelled.  For this reason, freezable kernelthreads must call try_to_freeze() somewhere or use one of thewait_event_freezable() and wait_event_freezable_timeout() macros.After the system memory state has been restored from a hibernation image anddevices have been reinitialized, the function thaw_processes() is called inorder to clear the PF_FROZEN flag for each frozen task.  Then, the tasks thathave been frozen leave refrigerator() and continue running.III. Which kernel threads are freezable?Kernel threads are not freezable by default.  However, a kernel thread may clearPF_NOFREEZE for itself by calling set_freezable() (the resetting of PF_NOFREEZEdirectly is strongly discouraged).  From this point it is regarded as freezableand must call try_to_freeze() in a suitable place.IV. Why do we do that?Generally speaking, there is a couple of reasons to use the freezing of tasks:1. The principal reason is to prevent filesystems from being damaged afterhibernation.  At the moment we have no simple means of checkpointingfilesystems, so if there are any modifications made to filesystem data and/ormetadata on disks, we cannot bring them back to the state from before themodifications.  At the same time each hibernation image contains somefilesystem-related information that must be consistent with the state of theon-disk data and metadata after the system memory state has been restored fromthe image (otherwise the filesystems will be damaged in a nasty way, usuallymaking them almost impossible to repair).  We therefore freeze tasks that mightcause the on-disk filesystems' data and metadata to be modified after thehibernation image has been created and before the system is finally powered off.The majority of these are user space processes, but if any of the kernel threadsmay cause something like this to happen, they have to be freezable.2. Next, to create the hibernation image we need to free a sufficient amount ofmemory (approximately 50% of available RAM) and we need to do that beforedevices are deactivated, because we generally need them for swapping out.  Then,after the memory for the image has been freed, we don't want tasks to allocateadditional memory and we prevent them from doing that by freezing them earlier.[Of course, this also means that device drivers should not allocate substantialamounts of memory from their .suspend() callbacks before hibernation, but thisis e separate issue.]3. The third reason is to prevent user space processes and some kernel threadsfrom interfering with the suspending and resuming of devices.  A user spaceprocess running on a second CPU while we are suspending devices may, forexample, be troublesome and without the freezing of tasks we would need somesafeguards against race conditions that might occur in such a case.Although Linus Torvalds doesn't like the freezing of tasks, he said this in oneof the discussions on LKML (http://lkml.org/lkml/2007/4/27/608):"RJW:> Why we freeze tasks at all or why we freeze kernel threads?Linus: In many ways, 'at all'.I _do_ realize the IO request queue issues, and that we cannot actually dos2ram with some devices in the middle of a DMA.  So we want to be able toavoid *that*, there's no question about that.  And I suspect that stoppinguser threads and then waiting for a sync is practically one of the easierways to do so.So in practice, the 'at all' may become a 'why freeze kernel threads?' andfreezing user threads I don't find really objectionable."Still, there are kernel threads that may want to be freezable.  For example, ifa kernel that belongs to a device driver accesses the device directly, it inprinciple needs to know when the device is suspended, so that it doesn't try toaccess it at that time.  However, if the kernel thread is freezable, it will befrozen before the driver's .suspend() callback is executed and it will bethawed after the driver's .resume() callback has run, so it won't be accessingthe device while it's suspended.4. Another reason for freezing tasks is to prevent user space processes fromrealizing that hibernation (or suspend) operation takes place.  Ideally, userspace processes should not notice that such a system-wide operation has occurredand should continue running without any problems after the restore (or resumefrom suspend).  Unfortunately, in the most general case this is quite difficultto achieve without the freezing of tasks.  Consider, for example, a processthat depends on all CPUs being online while it's running.  Since we need todisable nonboot CPUs during the hibernation, if this process is not frozen, itmay notice that the number of CPUs has changed and may start to work incorrectlybecause of that.V. Are there any problems related to the freezing of tasks?Yes, there are.First of all, the freezing of kernel threads may be tricky if they depend oneon another.  For example, if kernel thread A waits for a completion (in theTASK_UNINTERRUPTIBLE state) that needs to be done by freezable kernel thread Band B is frozen in the meantime, then A will be blocked until B is thawed, whichmay be undesirable.  That's why kernel threads are not freezable by default.Second, there are the following two problems related to the freezing of userspace processes:1. Putting processes into an uninterruptible sleep distorts the load average.2. Now that we have FUSE, plus the framework for doing device drivers inuserspace, it gets even more complicated because some userspace processes arenow doing the sorts of things that kernel threads do(https://lists.linux-foundation.org/pipermail/linux-pm/2007-May/012309.html).The problem 1. seems to be fixable, although it hasn't been fixed so far.  Theother one is more serious, but it seems that we can work around it by usinghibernation (and suspend) notifiers (in that case, though, we won't be able toavoid the realization by the user space processes that the hibernation is takingplace).There are also problems that the freezing of tasks tends to expose, althoughthey are not directly related to it.  For example, if request_firmware() iscalled from a device driver's .resume() routine, it will timeout and eventuallyfail, because the user land process that should respond to the request is frozenat this point.  So, seemingly, the failure is due to the freezing of tasks.Suppose, however, that the firmware file is located on a filesystem accessibleonly through another device that hasn't been resumed yet.  In that case,request_firmware() will fail regardless of whether or not the freezing of tasksis used.  Consequently, the problem is not really related to the freezing oftasks, since it generally exists anyway.A driver must have all firmwares it may need in RAM before suspend() is called.If keeping them is not practical, for example due to their size, they must berequested early enough using the suspend notifier API described in notifiers.txt.