CMPSC 447: Software Security, Spring 2026

Experiencing Format String Attacks

This project should be performed on machines in the CSE Linux Lab: e5-cse-135-XX.cse.psu.edu, where XX is a number between 01 and about 28. You can either use those machines by going to the CSE Linux lab, or you can access them remotely. For remote access, you will need to first vpn into the PSU network before being able to access those machines. We will test your results on those machines.

Part 1

Read the document Exploiting Format String Vulnerabilities by scut / team teso. Reading the first 3 sections should be sufficient for this homework.

Part 2 (10 points)

We continue our study of the lucky program in the last homework. For the convenience of performing a format string attack, we have modified the program slightly, as shown below.

#include <stdio.h>
#include <sys/time.h>
#include <stdlib.h>


char Password[100]; // Memory storage for the password

char goodPassword() {
  int good='N';
  char *Prompt="Password=";
  int *p = &good;

  fgets(Password, sizeof(Password), stdin); // Get input from keyboard

  printf(Prompt);
  printf(Password);
  printf("\n");

  return (char)(*p);
}

int main() {
  struct timeval t;

  printf("Enter your password:");

  char r = goodPassword();
  printf("r=%c\n", r);

  if (r == 'Y') {
    gettimeofday(&t, 0);
    srand((unsigned int) t.tv_usec);
    printf("Your lucky number today is %d!\n", rand()%100);
  }
  else {
    printf("No lucky number for you today.\n");
    exit(-1);
  }

  return 0;
}

Similar to the previous version, the program refuses to generate a lucky number. What is different is that it uses fgets to get the user password instead of the insecure gets. The fgets function takes the size of the Password buffer as the bound of the number of characters to read (more accurately, if the buffer size is n, fgets will read at most n-1 characters and will terminate the buffer with a null byte). Therefore, there is no possibility of overflowing the Password buffer through the fgets function.

However, there is a format-string vulnerability in the program. The program tries to echo the password in the goodPassword function, but it uses printf in an insecure way. We'd like to take advantage of this vulnerability so that the program generates lucky numbers for us.

Do this homework in the following steps:

(1 point) Pinpoint which statement in the program has a format-string vulnerability. Explain why.
(2 points) Try the password "%16lx %16lx %16lx %16lx %16lx %16lx %16lx %16lx %16lx". What is the behavior of the program? Explain what happens.
(2 point) Try the password "dddd%n". What is the behavior of the program? Explain what happens.
(5 points) We'd like to use a password of the form "%?c%?c...%?c%n" to attack the program. The password starts with a series of "%?c", where ? stands for a number we need to figure out. The purpose of this password is to 1) move the internal stack pointer (not rsp) of printf to point to the local variable p in goodPassword; 2) use the series of "%?c" to control the number of characters to print so that the number exactly equals the ASCII value of character 'Y'; 3) use "%n" to write 'Y' to memory location with the address in p. In the last step, since p contains the address of variable good, the effect is to change the value of good from 'N' to 'Y'. Given this attack, your job is to figure out the number of "%?c" at the beginning of this password, and a number to replace "?" in each "$?c".

Notes

Compile the program by "gcc lucky.c -o lucky".
"gcc -S lucky.c -o lucky.s" compiles a C program to assembly code.
"gcc -ggdb lucky.c -o lucky" produces an executable with debugging information, which is used by gdb.
gdb might be useful for Question 4. You may find ddd easier to use since it has a GUI interface. Alternatively, manually inserting printf calls into the program might be sufficient to figure out which stack slot holds variable p and other information.

What you need to submit

Submit your work through Canvas. The deadline is Feb 10th. The submission should be a text file or a PDF file that contains the answers to each step and also an explanation of the process you took to get your answers and how your answers work to achieve the desired results. Note that it's not enough to just give answers, you need to explain in detail how you got the answers and why they work, especially for Question 2 to 4 in Part 2.