CMPSC 311, The Posix Standard

CMPSC 311, Introduction to Systems Programming

The POSIX Standard
The Single UNIX Specification

Reading

CS:APP, p. 15, Aside, Unix and Posix
APUE, Sections 2.1-2.4, 2.7-2.10.

The Posix Standard describes the behavior of a computer system as seen through a particular set of data types, function interfaces and system utilities.

The Standard describes an interface, not an implementation.
There is no particular distinction between system functions and library functions, as the C99 libraries are included with the Posix libraries.

"The basic goal was to promote portability of application programs across UNIX system environments by developing a clear, consistent, and unambiguous standard for the interface specification of a portable operating system based on the UNIX system documentation. This standard codifies the common, existing definition of the UNIX system."

The official designation of the current standard is

IEEE Standard 1003.1^TM, 2008 Edition

also called POSIX.1-2008

International Standard ISO/IEC 9945:2009
(the core of) the Single UNIX Specification, Version 4

The Open Group Technical Standard, Base Specifications, Issue 7

This is one set of documents with multiple sponsors.
The standard is produced by the IEEE Standards Association and The Open Group.
The standard is maintained by The Austin Common Standards Revision Group.

The first meeting of the group was held at IBM's offices in Austin, TX, in 1998.
The most recent revisions were in 2001 (complete review), 2002 and 2004 (partial reviews for corrections and clarification).
The revision process for IEEE Std 1003.1-2008 began in Feb. 2006. The final draft of Issue 7 was dated May 2008, approved by The Open Group in July and by the IEEE in September, and published in Dec. 2008. The ISO/IEC version was published in Sep. 2009.
Technical Corrigendum 1 is in process, 2010-2013.

IEEE POSIX^® = Portable Operating System Interface

originally, a family of IEEE Standards published in pieces over time since 1988
The official title of the standard is IEEE Std 1003.1™-2008, “Standard for Information Technology — Portable Operating System Interface (POSIX^®)”.
The official sponsors of the standard are The Portable Applications Standards Committee of the IEEE Computer Society and The Open Group.

IEEE = Institute of Electrical and Electronics Engineers

formed in 1963 by the merger of the American Institute of Electrical Engineers (founded in 1884), and the Institute of Radio Engineers (founded in 1912)

Roughly speaking, wires and wireless, still the same issues.

IEEE Standards are widely accepted in the technical community, but they are voluntary and do not carry the legal weight of ISO/IEC Standards. The standards are reviewed every five years.
IEEE Standards Association
The Portable Applications Standards Committee of the IEEE Computer Society

PASC is now largely inactive, succeeded by the Austin Group

The Open Group

formed in 1996 by the merger of X/Open and the Open Software Foundation
now the legal owner of the UNIX trademark that originated with AT&T
http://www.unix.org
history of the standardization process

SUS = Single UNIX Specification

one standard with options, especially the X/Open System Interface (XSI)
SUS Version 4, main page and overview
FAQs, including background information
The Authorized Guide to the Single UNIX Specification, Version 4 (free, but registration is required)

Why do we need a standard for an operating system?

"POSIX.1-2008 defines a standard operating system interface and environment, including a command interpreter (or shell), and common utility programs to support applications portability at the source code level. It is intended to be used by both applications developers and system implementors."
"POSIX.1-2008 describes the external characteristics and facilities that are of importance to applications developers, rather than the internal construction techniques employed to achieve these capabilities. Special emphasis is placed on those functions and facilities that are needed in a wide variety of commercial applications."
For more issues, refer to the previous discussion of the C Standard.

Do I actually need to read the Posix Standard?

That's not the place to start learning about UNIX, but if you plan to write system programs, or if you plan to decide if someone else's system programs are correct before you ship them or use them, or if you expect to use system functions and system programs properly, then you need to know about the standard.

The 2008 version is four volumes in one PDF, 12 MB, 3,874 pages.

"The intended audience for POSIX.1-2008 is all persons concerned with an industry-wide standard operating system based on the UNIX system. This includes at least four groups of people:
1. Persons buying hardware and software systems,
2. Persons managing companies that are deciding on future corporate computing directions,
3. Persons implementing operating systems, and especially,
4. Persons developing applications where portability is an objective."

POSIX or Posix?

POSIX is correct, Posix is acceptable as far as this course is concerned.
IEEE POSIX® is the registered trademark.
There is a certification process, so only systems that have "passed the test" can legally claim to be POSIX-compliant.
"The name POSIX was suggested by Richard Stallman. It is expected to be pronounced pahz-icks, as in positive, not poh-six, or other variations. The pronunciation has been published in an attempt to promulgate a standardized way of referring to a standard operating system interface."

UNIX or Unix?

It depends on how precise you want to be. Most people tend to be sloppy about it.
UNIX® is the registered trademark, now owned by The Open Group.
UNIX refers to an operating system interface that has been certified as being compliant with the Single UNIX Specification.

UNIX requires the XSI option (described later).
The current list of certified UNIX 03 Registered Products is here. These are "UNIX branded".

Unix refers to an operating system that claims to be UNIX, or to an abstraction of the concepts underlying these systems.
Unix-like or UNIX-like refers to an operating system that looks like UNIX at some meaningful level (for example, Linux or Mac OS X).
*nix is a common designator with even less precision.

The use of an IEEE standard is voluntary, but since The Open Group owns the rights to the UNIX name, you need their certification to call your operating system UNIX, or UNIX-compatible.

See APUE Sections 2.2, 2.3, 2.4 for

the history of the Posix Standard, and a description of the headers (include files)
the history of the various Unix families
notes of standards vs. implementations

There are four volumes in the Posix Standard:

Base Definitions

general terms and concepts, utility conventions, C language header file specifications

System Interfaces

C language function prototypes, with descriptions of behavior, portability, error handling, and error recovery

Shell and Utilities

names, options and operands of utility programs, with descriptions of behavior, portability, error handling, and error recovery

Rationale (informative)

explanations and historical information

These areas are not included (among others)

Graphics interfaces
Database management system interfaces
Object or binary code portability
System configuration and resource availability

To access the standard ...

To access the standard as HTML files, go here (The Open Group). This is the most convenient method, but the formatting is sometimes a little strange, and you need to register first.

Or, here.

To access the standard as PDF files, from a PSU system, or if you are an IEEE member, go to IEEE Xplore

Click on Standards under Browse, then do a keyword search with "1003".
These are the POSIX standards, 1988-2008. The most recent version is numbered 1003.1-2008.
Penn State paid a fee to the IEEE, otherwise you need to pay it.

From a non-PSU system, via the PSU library, go to IEEE Xplore (a different link)

You will prompted for your PSU Access Account ID.

If you are running Linux, the man pages have excerpts from the 2003 version of the standard in section P (for example, man -a ffs).

This might mention a link http://www.open-group.org/unix/online.html but the hyphen is to split the word opengroup across two lines, and should not appear in the link as you use it.
NetBSD and FreeBSD should also provide this information.

Some older designations that you may see mentioned in various places,

1003.1-1988 was the original system interfaces standard (Posix.1)
1003.1-1990 = ISO/IEC 9945-1:1990, with a later version in 1996 as ANSI/IEEE 1003.1-1996 = ISO/IEC 9945-1:1996
1003.2-1992 = ISO/IEC 9945-2:1993 was the shell and utilities standard (Posix.2)
2003.1-1992 was the conformance testing standard
1003.0-1995 = ISO/IEC TR 14252 was a guidebook, withdrawn in Jan. 2002
1003.1-2001 is the previous base document, for which corrections were issued in 2002 and 2004.
There is a longer list on APUE pp. 27-28, but there were many others.

There has also been a series of IEEE Posix standards for interfaces to the Ada and Fortran programming languages, and there is a current project (P1003.27) to produce Posix interfaces for C++. There are related standards for Realtime and Embedded Application Support (1003.13-2003) and Device Control Application Program Interface (1003.26-2003).

Of historical interest,

ISO/IEC JTC1/SC22/WG15 - POSIX, disbanded in 2004, replaced by the SC22 Posix Advisory Group
X/Open Portability Guide, Issue 1, 1985; Issue 2, 1987; Issue 3, 1988, 1989, 1992; Issue 4, 1992, 1994, now by The Open Group; Issue 5, 1997

Of current interest,

WG, 1003.27, for POSIX-C++ Binding

https://www.redhat.com/mailman/listinfo/posix-c++-wg

Version Test Macros

How can you tell which version of the Posix interface you have?

These use the symbolic constants #define'd by <unistd.h>. There are many more of these which describe the various options of the standard.

Here is an example from the Posix Standard, which uses a compile-time version test

#if _POSIX_VERSION >=
        200112L

  /* Use the newer function that copes with large
        files. */

off_t pos=ftello(fp);
#else

  /* Either this is an old version of POSIX, or
        _POSIX_VERSION is

   * not even defined, so use the traditional
        function.

           */

long pos=ftell(fp);
#endif

This example uses a runtime version test,

if (sysconf(_SC_VERSION) <
        200112L) {

  fprintf(stderr, "POSIX.1-2001 system required,
        terminating.\n");

exit(1);
}

Posix Vocabulary

Some features are marked legacy or obsolescent. These are allowed by the current standard, but will probably be withdrawn in future versions. You should avoid using these features in new programs, and if you find them in old programs (that you have the authority to rewrite) then you should replace them.

For example, replace bcopy() with memcpy().

The term implementation-defined is used in the same way as in the C Standard. In the case of some possibly-defined constants, the constant might not be defined at all, but in any case the documentation should clearly state what has been implemented or not.

The terms undefined and unspecified relate to values or behaviors associated with invalid or valid programs or data, respectively. Portable programs should not depend on undefined or unspecified features.

There are various levels of requirements placed by the standard, depending on the verb that is used (some of this comes directly from the ISO/IEC requirements). "Can" refers to optional application behavior. "May" refers to optional implementation behavior.

shall

For an implementation, describes a feature or behavior that is mandatory. An application can rely on the existence of the feature or behavior.

For an application or user, describes a behavior that is mandatory.

For an implementation, describes a feature or behavior that is mandatory. An application can rely on the existence of the feature or behavior.
For an application or user, describes a permissible optional feature or behavior.

For an implementation, describes a feature or behavior that is optional. An application should not rely on the existence of the feature or behavior. An application that relies on such a feature or behavior cannot be assured to be portable across conforming implementations.

To avoid ambiguity, the opposite of may is expressed as need not, instead of may not.

should

For an implementation, describes a feature or behavior that is recommended but not mandatory. An application should not rely on the existence of the feature or behavior. An application that relies on such a feature or behavior cannot be assured to be portable across conforming implementations.

For an application, describes a feature or behavior that is recommended programming practice for optimum portability.

A conforming implementation shall meet all of the following criteria: (lightly edited from the Standard)

The system shall support all utilities, functions, and facilities defined within IEEE Std 1003.1-2008 that are required for POSIX conformance. These interfaces shall support the functional behavior described herein.
The system may support one or more options as described under Section 2.1.5. When an implementation claims that an option is supported, all of its constituent parts shall be provided.
The system may support the X/Open System Interface Extension (XSI) as described under Section 2.1.4.
The system may provide additional utilities, functions, or facilities not required by IEEE Std 1003.1-2008.

Non-standard extensions of the utilities, functions, or facilities specified in IEEE Std 1003.1-2008 should be identified as such in the system documentation.
Non-standard extensions, when used, may change the behavior of utilities, functions, or facilities defined by IEEE Std 1003.1-2008. The conformance document shall define an environment in which an application can be run with the behavior specified by IEEE Std 1003.1-2008. In no case shall such an environment require modification of a Strictly Conforming POSIX Application (see Section 2.2.1).

A conforming POSIX application must be written in C, or use only the standard utilities (it could be a shell script).

A strictly conforming POSIX application

requires only the standard facilities,
accepts any implementation-defined or unspecified behavior,
does not perform any actions that produce undefined results,
accepts any C- or Posix-standard-defined symbolic constants within the defined ranges, but does not rely on values outside the most restrictive range,
does not use obsolescent facilities,
does not produce any output dependent on any behavior described in the C99 standard as unspecified, undefined, or implementation-defined, unless the System Interfaces volume of the Posix Standard specifies the behavior
defines _POSIX_C_SOURCE to be 200809L before any header is included.

A strictly conforming XSI application would also define _XOPEN_SOURCE to be 700 before any header is included.

Use the values 200112L and 600 for the 2001 edition of the standard.
Use the values 200809L and 700 for the 2008 edition of the standard.

Feature Test Macros

To ensure that your code uses only the standard features from Posix or XSI, compile it on Solaris as

c99 -D_POSIX_C_SOURCE=200112L
        prog.c

c99 -D_XOPEN_SOURCE=600 prog.c

The first version excludes implementation-defined definitions or limits. The second version includes all XSI features, and also sets _POSIX_C_SOURCE to 200112L. Use

gcc
      -std=c99

instead of c99 on Linux or Mac OS X.

Using these two macros would be an easy way to assess how close to "maximally portable" your program might be.

Use the values 200112L and 600 for the 2001 edition of the standard.
Use the values 200809L and 700 for the 2008 edition of the standard.

Headers and data types defined in the Posix Standard

The headers marked C99 in the Option column are adopted by Posix from the C Standard, either directly or with extensions. If an entry in the Option column is blank, that header is required by Posix but not by C99. Refer to APUE, Fig. 2.5 for a more complete list of option codes, and which ones are required for the Single UNIX Specification; of those in this table, they are THR (threads) and XSI (X/Open System Interface). See also, Posix Option Codes for the full list.

Header	Posix Standard description	Option	APUE
`<aio.h>`	asynchronous input and output	AIO
`<arpa/inet.h>`	definitions for internet operations		16.
`<assert.h>`	verify program assertion	C99
`<complex.h>`	complex arithmetic	C99
`<cpio.h>`	cpio archive values	XSI
`<ctype.h>`	character types	C99
`<dirent.h>`	format of directory entries		4.21
`<dlfcn.h>`	dynamic linking	XSI
`<errno.h>`	system error numbers	C99
`<fcntl.h>`	file control options		3.14
`<fenv.h>`	floating-point environment	C99
`<float.h>`	floating types	C99
`<fmtmsg.h>`	message display structures	XSI
`<fnmatch.h>`	filename-matching types
`<ftw.h>`	file tree traversal	XSI	4.21
`<glob.h>`	pathname pattern-matching types
`<grp.h>`	group structure		6.4
`<iconv.h>`	codeset conversion facility	XSI
`<inttypes.h>`	fixed size integer types	C99
`<iso646.h>`	alternative spellings	C99
`<langinfo.h>`	language information constants	XSI
`<libgen.h>`	definitions for pattern matching functions	XSI
`<limits.h>`	implementation-defined constants	C99
`<locale.h>`	category macros	C99
`<math.h>`	mathematical declarations	C99
`<monetary.h>`	monetary types	XSI
`<mqueue.h>`	message queues	MSG
`<ndbm.h>`	definitions for ndbm database operations	XSI
`<net/if.h>`	sockets local interfaces		16.
`<netdb.h>`	definitions for network database operations
`<netinet/in.h>`	Internet address family		16.3
`<netinet/tcp.h>`	definitions for the Internet Transmission Control Protocol (TCP)
`<nl_types.h>`	data types	XSI
`<poll.h>`	definitions for the poll( ) function	XSI	14.5.2
`<pthread.h>`	threads	THR	11., 12.
`<pwd.h>`	password structure		6.2
`<regex.h>`	regular expression matching types
`<sched.h>`	execution scheduling	PS
`<search.h>`	search tables	XSI
`<semaphore.h>`	semaphores	SEM
`<setjmp.h>`	stack environment declarations	C99
`<signal.h>`	signals	C99
`<spawn.h>`	spawn	SPN
`<stdarg.h>`	handle variable argument list	C99
`<stdbool.h>`	boolean type and values	C99
`<stddef.h>`	standard type definitions	C99
`<stdint.h>`	integer types	C99
`<stdio.h>`	standard buffered input/output	C99
`<stdlib.h>`	standard library definitions	C99
`<string.h>`	string operations	C99
`<strings.h>`	string operations	XSI
`<stropts.h>`	STREAMS interface	XSR	14.4
`<sys/ipc.h>`	XSI interprocess communication access structure	XSI	15.6
`<sys/mman.h>`	memory management declarations
`<sys/msg.h>`	memory management declarations	XSI	15.7
`<sys/resource.h>`	definitions for XSI resource operations	XSI	7.11
`<sys/select.h>`	select types		14.5.1
`<sys/sem.h>`	XSI semaphore facility	XSI	15.8
`<sys/shm.h>`	XSI shared memory facility	XSI	15.9
`<sys/socket.h>`	main sockets header		16.
`<sys/stat.h>`	data returned by the stat( ) function		4.
`<sys/statvfs.h>`	VFS File System information structure	XSI
`<sys/time.h>`	time types	XSI
`<sys/timeb.h>`	additional definitions for date and time	XSI
`<sys/times.h>`	file access and modification times structure		8.16
`<sys/types.h>`	data types		2.8
`<sys/uio.h>`	definitions for vector I/O operations	XSI	14.7
`<sys/un.h>`	definitions for UNIX domain sockets		17.3
`<sys/utsname.h>`	system name structure		6.9
`<sys/wait.h>`	declarations for waiting		8.6
`<syslog.h>`	definitions for system error logging	XSI	13.4
`<tar.h>`	extended tar definitions
`<termios.h>`	define values for termios (terminal I/O)		18.
`<tgmath.h>`	type-generic macros	C99
`<time.h>`	time types	C99
`<trace.h>`	tracing	TRC
`<ucontext.h>`	user context	XSI
`<ulimit.h>`	ulimit commands	XSI
`<unistd.h>`	standard symbolic constants and types		various
`<utime.h>`	access and modification times structure		4.19
`<utmpx.h>`	user accounting database definitions	XSI
`<wchar.h>`	wide-character handling	C99
`<wctype.h>`	wide-character classification and mapping utilities	C99
`<wordexp.h>`	word-expansion types

The following table is derived from the POSIX Standard, System Interfaces, Sec. 2.12, Data Types. It extends APUE Fig. 2.20, but it is still not a complete list of all types defined in the Standard (we will enlarge the table as time permits).

Again, recall the rule that the standard is an interface, not an implementation. The appropriate header is required to have a typedef, or struct or union declaration, which actually implements the type, and which follows rules set by the standard, but the implementation itself is not given by the standard.

Some types appear in more than one header, and most headers include other headers. Each entry in this table lists only one header, for convenience. For example, the pthread_t type is defined in <sys/types.h>, which is included by the more obvious choice <pthread.h>.

Header	Type	Description
`<termios.h>`	`cc_t`	Type used for terminal special characters.
`<sys/types.h>`	`clock_t`	Integer or real-floating type used for processor times, as defined in the ISO C standard.
`<sys/types.h>`	`clockid_t`	Used for clock ID type in some timer functions.
`<sys/types.h>`	`dev_t`	Arithmetic type used for device numbers.
`<dirent.h>`	`DIR`	Type representing a directory stream.
`<stdlib.h>`	`div_t`	Structure type returned by the `div()` function.
`<stdio.h>`	`FILE`	Structure containing information about a file.
`<glob.h>`	`glob_t`	Structure type used in pathname pattern matching.
`<stdio.h>`	`fpos_t`	Type containing all information needed to specify uniquely every position within a file.
`<sys/types.h>`	`gid_t`	Integer type used for group IDs.
`<ioconv.h>`	`iconv_t`	Type used for conversion descriptors.
`<sys/types.h>`	`id_t`	Integer type used as a general identifier; can be used to contain at least the largest of a `pid_t`, `uid_t`, or `gid_t`.
`<sys/types.h>`	`ino_t`	Unsigned integer type used for file serial numbers.
`<sys/types.h>`	`key_t`	Arithmetic type used for XSI interprocess communication.
`<stdlib.h>`	`ldiv_t`	Structure type returned by the `ldiv()` function.
`<sys/types.h>`	`mode_t`	Integer type used for file attributes.
`<mqueue.h>`	`mqd_t`	Used for message queue descriptors.
`<poll.h>`	`nfds_t`	Integer type used for the number of file descriptors.
`<sys/types.h>`	`nlink_t`	Integer type used for link counts.
`<sys/types.h>`	`off_t`	Signed integer type used for file sizes.
`<sys/types.h>`	`pid_t`	Signed integer type used for process and process group IDs.
`<sys/types.h>`	`pthread_attr_t`	Used to identify a thread attribute object.
`<sys/types.h>`	`pthread_cond_t`	Used for condition variables.
`<sys/types.h>`	`pthread_condattr_t`	Used to identify a condition attribute object.
`<sys/types.h>`	`pthread_key_t`	Used for thread-specific data keys.
`<sys/types.h>`	`pthread_mutex_t`	Used for mutexes.
`<sys/types.h>`	`pthread_mutexattr_t`	Used to identify a mutex attribute object.
`<sys/types.h>`	`pthread_once_t`	Used for dynamic package initialization.
`<sys/types.h>`	`pthread_rwlock_t`	Used for read-write locks.
`<sys/types.h>`	`pthread_rwlockattr_t`	Used for read-write lock attributes.
`<sys/types.h>`	`pthread_t`	Used to identify a thread.
`<stddef.h>`	`ptrdiff_t`	Signed integer type of the result of subtracting two pointers.
`<regex.h>`	`regex_t`	Structure type used in regular expression matching.
`<regex.h>`	`regmatch_t`	Structure type used in regular expression matching.
`<sys/resource.h>`	`rlim_t`	Unsigned integer type used for limit values, to which objects of type `int` and `off_t` can be cast without loss of value.
`<semaphore.h>`	`sem_t`	Type used in performing semaphore operations.
`<signal.h>`	`sig_atomic_t`	Integer type of an object that can be accessed as an atomic entity, even in the presence of asynchronous interrupts.
`<signal.h>`	`sigset_t`	Integer or structure type of an object used to represent sets of signals.
`<stddef.h>`	`size_t`	Unsigned integer type used for size of objects.
`<termios.h>`	`speed_t`	Type used for terminal baud rates.
`<sys/types.h>`	`ssize_t`	Signed integer type used for a count of bytes or an error indication.
`<sys/types.h>`	`suseconds_t`	Signed integer type used for time in microseconds.
`<termios.h>`	`tcflag_t`	Type used for terminal modes.
`<sys/types.h>`	`time_t`	Integer or real-floating type used for time in seconds, as defined in the ISO C standard.
`<sys/types.h>`	`timer_t`	Used for timer ID returned by the `timer_create()` function.
`<sys/types.h>`	`uid_t`	Integer type used for user IDs.
`<sys/types.h>`	`useconds_t`	Unsigned integer type used for time in microseconds.
`<stdarg.h>`	`va_list`	Type used for traversing variable argument lists.
`<stddef.h>`	`wchar_t`	Integer type whose range of values can represent distinct codes for all members of the largest extended character set specified by the supported locales.
`<wctype.h>`	`wctype_t`	Scalar type which represents a character class descriptor.
`<wchar.h>`	`wint_t`	Integer type capable of storing any valid value of `wchar_t` or `WEOF`.
`<wordexp.h>`	`wordexp_t`	Structure type used in word expansion.

References, not previously mentioned

Unless otherwise stated, quotes on this page are taken from the Posix Standard, 2008 edition.
Solaris, standards(5). Solaris 10 conforms to nearly everything, and is UNIX 03 Branded. For some utilities, you may need to supply a full pathname to get a particular version of the utility adhering to a particular version of the various standards.

Linux

The Linux Foundation, The Linux Standard Base. This plays a role similar to The Open Group and the Single UNIX Specification, but with more emphasis on applications and distributions.
The report "Linux Kernel Development: How Fast it is Going, Who is Doing It, What They are Doing, and Who is Sponsoring It" (April 2008, August 2009, Dec. 2010, March 2012) gives you some good perspective on the problems and successes of open source development. Highly recommended.

Last revised, 22 Jan. 2013