You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-23 - 16:41

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot5s.osadl.org (updated Sun Nov 23, 2025 12:44:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1042899250,24cyclictest9610-21wc11:35:000
10437992423,1cyclictest0-21swapper/211:32:152
10437992421,2cyclictest7922-21cron10:25:002
1043399240,2cyclictest26973-21sh08:50:001
10428992423,1cyclictest24482-21apt-get12:07:160
10439992323,0cyclictest0-21swapper/308:57:163
1043999230,22cyclictest19202-21cron11:55:003
1043799230,0cyclictest0-21swapper/208:57:162
10433992321,1cyclictest29996-21cron11:10:001
1043399230,22cyclictest4246-21perf09:10:001
10439992220,1cyclictest19355-21cron08:35:003
1043999220,21cyclictest0-21swapper/310:22:213
1043999220,21cyclictest0-21swapper/310:12:163
1043999220,1cyclictest0-21swapper/311:05:003
10437992221,1cyclictest1149-21apt-get10:12:162
10437992221,1cyclictest10045-21apt-get11:37:162
10437992220,1cyclictest6654-21grep09:15:002
10437992220,1cyclictest0-21swapper/212:07:182
10437992220,1cyclictest0-21swapper/209:45:012
1043799220,21cyclictest0-21swapper/212:22:172
1043799220,21cyclictest0-21swapper/211:27:172
1043799220,21cyclictest0-21swapper/210:27:152
1043799220,1cyclictest313-21systemd-journal08:45:002
10433992222,0cyclictest0-21swapper/109:40:011
10433992222,0cyclictest0-21swapper/107:12:151
10433992220,1cyclictest31844-21turbostat09:00:001
10433992220,1cyclictest19245-21munin-run11:55:001
1043399220,21cyclictest29454-21perf08:55:011
1043399220,21cyclictest29329-21apt-get12:17:161
1043399220,21cyclictest15600-21apt-get10:42:181
1043399220,21cyclictest1293-21apt-get12:25:011
1043399220,21cyclictest11900-21apt-get09:27:141
1043399220,21cyclictest0-21swapper/110:07:161
1043399220,1cyclictest4678-21apt-get09:12:151
10428992220,1cyclictest21804-21sendmail-msp08:40:000
1042899220,21cyclictest24896-21turbostat.cron07:40:000
1042899220,21cyclictest0-21swapper/011:52:150
10439992120,0cyclictest23184-21apt-get10:57:173
10439992120,0cyclictest22830-21apt-get07:37:143
10439992119,1cyclictest31175-21apt-get10:07:163
10439992118,2cyclictest32418-21latency_hist11:15:013
1043999210,21cyclictest0-21swapper/312:07:173
1043999210,21cyclictest0-21swapper/307:27:163
1043999210,20cyclictest0-21swapper/311:47:173
1043999210,1cyclictest17618-21munin-run07:25:003
1043999210,0cyclictest0-21swapper/312:02:173
1043999210,0cyclictest0-21swapper/310:42:173
1043999210,0cyclictest0-21swapper/308:52:173
1043999210,0cyclictest0-21swapper/308:07:143
10437992120,0cyclictest0-21swapper/212:12:172
1043799210,20cyclictest0-21swapper/212:32:152
1043799210,20cyclictest0-21swapper/211:52:162
1043799210,20cyclictest0-21swapper/207:57:142
1043799210,20cyclictest0-21swapper/207:32:152
1043799210,1cyclictest30596-21apt-get07:52:142
1043799210,1cyclictest0-21swapper/207:45:022
10433992121,0cyclictest21509-21apt-get09:47:151
10433992120,1cyclictest5400-21apt-get08:07:141
10433992120,1cyclictest0-21swapper/112:32:161
10433992119,1cyclictest6926-21apt-get12:37:171
1043399210,21cyclictest0-21swapper/110:45:011
1043399210,21cyclictest0-21swapper/108:02:141
1043399210,20cyclictest0-21swapper/112:07:171
1043399210,20cyclictest0-21swapper/107:57:141
1043399210,0cyclictest21780-21apt-get08:40:021
1043399210,0cyclictest0-21swapper/111:37:161
10428992121,0cyclictest7799-21apt-get08:12:170
10428992121,0cyclictest12472-21apt-get11:42:170
10428992121,0cyclictest0-21swapper/012:02:150
10428992121,0cyclictest0-21swapper/011:07:160
10428992119,1cyclictest4977-21cat08:05:000
1042899210,21cyclictest0-21swapper/008:22:160
1042899210,20cyclictest0-21swapper/011:22:170
1042899210,1cyclictest12194-21sendmail-msp08:20:000
1042899210,0cyclictest0-21swapper/010:27:160
1042899210,0cyclictest0-21swapper/010:07:160
1042899210,0cyclictest0-21swapper/009:47:150
10439992020,0cyclictest0-21swapper/311:32:163
10439992020,0cyclictest0-21swapper/309:42:143
10439992020,0cyclictest0-21swapper/309:32:163
10439992019,1cyclictest27307-21apt-get07:45:023
10439992019,1cyclictest0-21swapper/308:37:153
10439992019,0cyclictest8373-21apt-get10:27:163
10439992019,0cyclictest32292-21apt-get09:02:153
10439992019,0cyclictest0-21swapper/310:37:153
10439992018,1cyclictest3563-21apt-get10:17:173
1043999200,20cyclictest0-21swapper/307:52:143
1043999200,0cyclictest0-21swapper/312:25:013
1043999200,0cyclictest0-21swapper/309:22:153
1043999200,0cyclictest0-21swapper/307:32:153
10437992019,1cyclictest27427-21chrt08:52:172
10437992019,0cyclictest0-21swapper/207:17:162
10437992018,2cyclictest12189-21perf08:20:002
10437992018,1cyclictest5523-21sh10:20:002
1043799200,20cyclictest0-21swapper/208:12:172
1043799200,0cyclictest13176-21apt-get10:37:152
1043799200,0cyclictest0-21swapper/212:37:162
1043799200,0cyclictest0-21swapper/210:57:172
1043799200,0cyclictest0-21swapper/209:32:152
1043799200,0cyclictest0-21swapper/208:37:152
1043799200,0cyclictest0-21swapper/208:32:132
10433992020,0cyclictest0-21swapper/112:02:161
10433992020,0cyclictest0-21swapper/109:02:161
10433992020,0cyclictest0-21swapper/108:27:181
10433992019,1cyclictest2856-21apt-get11:22:171
10433992019,0cyclictest7078-21apt-get09:17:171
10433992019,0cyclictest28149-21apt-get07:47:161
10433992017,2cyclictest10322-21sh10:30:011
1043399200,20cyclictest0-21swapper/110:37:151
1043399200,20cyclictest0-21swapper/110:17:161
1043399200,20cyclictest0-21swapper/108:22:151
10428992020,0cyclictest0-21swapper/012:37:170
10428992020,0cyclictest0-21swapper/010:47:180
10428992020,0cyclictest0-21swapper/007:52:140
10428992020,0cyclictest0-21swapper/007:12:140
10428992019,1cyclictest17416-21apt-get08:32:150
10428992019,0cyclictest0-21swapper/012:32:180
1042899200,20cyclictest0-21swapper/009:07:140
1042899200,0cyclictest0-21swapper/012:25:010
10439991919,0cyclictest3004-21apt-get08:02:143
10439991919,0cyclictest0-21swapper/312:37:173
10439991919,0cyclictest0-21swapper/312:32:163
10439991919,0cyclictest0-21swapper/312:22:173
10439991919,0cyclictest0-21swapper/311:27:183
10439991919,0cyclictest0-21swapper/311:17:153
10439991919,0cyclictest0-21swapper/307:42:153
10439991918,1cyclictest0-21swapper/312:12:173
10439991917,1cyclictest6655-21cut09:15:003
10437991919,0cyclictest22082-21apt-get12:02:172
10437991919,0cyclictest10779-21apt-get10:32:142
10437991919,0cyclictest0-21swapper/212:27:152
10437991919,0cyclictest0-21swapper/208:07:152
10437991917,1cyclictest593-21dbus-daemon09:55:002
10437991917,1cyclictest31293-21idleruntime-cro12:20:002
1043799190,19cyclictest0-21swapper/207:27:162
10433991919,0cyclictest0-21swapper/111:57:161
10433991919,0cyclictest0-21swapper/111:27:181
10433991919,0cyclictest0-21swapper/110:52:181
10433991919,0cyclictest0-21swapper/110:32:151
10433991919,0cyclictest0-21swapper/110:22:201
10433991919,0cyclictest0-21swapper/109:32:161
10433991919,0cyclictest0-21swapper/107:52:141
10433991919,0cyclictest0-21swapper/107:27:161
10433991918,1cyclictest20432-21apt-get07:32:141
10433991918,0cyclictest0-21swapper/108:12:161
1043399190,19cyclictest0-21swapper/111:02:161
10428991919,0cyclictest0-21swapper/009:57:180
10428991919,0cyclictest0-21swapper/008:27:180
10428991919,0cyclictest0-21swapper/007:57:130
10428991919,0cyclictest0-21swapper/007:47:160
1042899190,0cyclictest0-21swapper/009:17:160
10439991818,0cyclictest535-21apt-get07:57:143
10439991818,0cyclictest0-21swapper/311:42:163
10439991818,0cyclictest0-21swapper/311:37:153
10439991818,0cyclictest0-21swapper/308:12:173
1043999180,17cyclictest10197-21apt-get08:17:173
10437991818,0cyclictest14870-21apt-get11:47:172
10437991818,0cyclictest0-21swapper/210:47:192
10437991818,0cyclictest0-21swapper/209:27:162
10437991818,0cyclictest0-21swapper/207:22:162
10433991818,0cyclictest0-21swapper/111:47:171
10433991818,0cyclictest0-21swapper/111:42:171
10433991818,0cyclictest0-21swapper/111:12:161
10433991818,0cyclictest0-21swapper/109:42:151
10433991818,0cyclictest0-21swapper/107:42:161
10433991818,0cyclictest0-21swapper/107:37:151
10433991818,0cyclictest0-21swapper/107:22:151
10433991817,0cyclictest0-21swapper/107:17:161
10428991818,0cyclictest28777-21apt-get10:02:150
10428991818,0cyclictest16698-21apt-get09:37:160
10428991818,0cyclictest10779-21apt-get10:32:150
10428991818,0cyclictest0-21swapper/012:17:160
10428991818,0cyclictest0-21swapper/010:22:200
10428991818,0cyclictest0-21swapper/008:40:020
10428991817,1cyclictest31752-21sshd12:22:170
10428991817,1cyclictest16273-21grep09:35:000
10428991817,0cyclictest0-21swapper/008:07:150
10428991816,1cyclictest18035-21apt-get07:27:150
10428991816,1cyclictest10045-21apt-get11:37:160
10439991717,0cyclictest0-21swapper/308:42:163
10439991715,1cyclictest23905-21apt-get09:52:143
10437991716,1cyclictest27577-21munin-run11:05:002
10433991717,0cyclictest0-21swapper/108:17:161
10433991715,1cyclictest0-21swapper/110:57:171
10428991717,0cyclictest30449-21apt-get11:12:170
10428991717,0cyclictest25355-21apt-get07:42:160
10428991717,0cyclictest0-21swapper/011:47:170
10428991717,0cyclictest0-21swapper/010:37:150
10428991717,0cyclictest0-21swapper/009:12:160
10428991717,0cyclictest0-21swapper/007:32:140
10428991716,1cyclictest0-21swapper/007:22:160
10439991616,0cyclictest0-21swapper/310:47:183
10439991616,0cyclictest0-21swapper/310:32:153
10439991616,0cyclictest0-21swapper/307:17:163
10437991616,0cyclictest379-21apt-get11:17:152
10437991616,0cyclictest12620-21apt-get08:22:152
10437991616,0cyclictest0-21swapper/211:22:162
10437991616,0cyclictest0-21swapper/211:12:162
10437991616,0cyclictest0-21swapper/209:47:152
10437991616,0cyclictest0-21swapper/209:02:162
10437991615,1cyclictest0-21swapper/207:12:152
10437991615,0cyclictest27998-21apt-get11:07:172
10437991615,0cyclictest0-21swapper/208:02:152
10437991614,1cyclictest0-21swapper/210:52:252
10437991614,1cyclictest0-21swapper/208:29:592
10433991616,0cyclictest0-21swapper/111:17:141
10433991616,0cyclictest0-21swapper/109:22:141
1043399160,15cyclictest27257-21cat12:12:161
10428991616,0cyclictest0-21swapper/010:52:180
10428991616,0cyclictest0-21swapper/009:42:150
10428991614,1cyclictest3239-21latency_hist10:14:590
10439991515,0cyclictest0-21swapper/309:27:163
10439991513,1cyclictest0-21swapper/311:55:213
1043999150,9cyclictest0-21swapper/309:07:153
1043999150,1cyclictest0-21swapper/308:27:233
1043999150,14cyclictest0-21swapper/309:59:223
10437991515,0cyclictest0-21swapper/209:07:162
10437991515,0cyclictest0-21swapper/207:42:162
1043799150,14cyclictest0-21swapper/209:59:552
10433991515,0cyclictest26310-21apt-get09:57:181
10433991515,0cyclictest0-21swapper/109:52:151
10428991515,0cyclictest9494-21apt-get09:22:150
10428991515,0cyclictest0-21swapper/011:17:150
10428991515,0cyclictest0-21swapper/009:52:150
10428991515,0cyclictest0-21swapper/009:27:160
10428991515,0cyclictest0-21swapper/008:47:170
10428991514,1cyclictest27563-21turbostat11:04:590
1042899150,14cyclictest0-21swapper/010:42:260
1042899150,14cyclictest0-21swapper/008:52:180
10439991414,0cyclictest0-21swapper/309:45:453
1043999140,6cyclictest578-21irqbalance10:50:453
1043999140,14cyclictest0-21swapper/311:05:063
1043999140,14cyclictest0-21swapper/308:45:193
1043999140,14cyclictest0-21swapper/308:21:083
1043999140,0cyclictest0-21swapper/310:00:273
1043999140,0cyclictest0-21swapper/309:35:133
1043999140,0cyclictest0-21swapper/309:17:083
1043999140,0cyclictest0-21swapper/307:10:533
10437991414,0cyclictest0-21swapper/211:41:252
1043799140,14cyclictest0-21swapper/210:40:372
1043799140,14cyclictest0-21swapper/210:00:402
1043799140,14cyclictest0-21swapper/209:35:292
1043799140,14cyclictest0-21swapper/208:49:402
1043799140,14cyclictest0-21swapper/207:35:432
1043799140,0cyclictest0-21swapper/211:55:402
1043799140,0cyclictest0-21swapper/209:20:032
1043799140,0cyclictest0-21swapper/209:16:512
10433991414,0cyclictest0-21swapper/111:30:121
10433991414,0cyclictest0-21swapper/108:30:321
1043399140,14cyclictest0-21swapper/112:20:291
1043399140,14cyclictest0-21swapper/108:36:021
1043399140,0cyclictest0-21swapper/110:10:451
10428991414,0cyclictest0-21swapper/010:57:170
1042899140,14cyclictest0-21swapper/012:10:170
1042899140,14cyclictest0-21swapper/011:55:200
1042899140,14cyclictest0-21swapper/010:15:480
1042899140,14cyclictest0-21swapper/007:15:180
1042899140,0cyclictest0-21swapper/011:27:100
1042899140,0cyclictest0-21swapper/009:00:020
10428991313,0cyclictest0-21swapper/008:55:180
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional