You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-14 - 12:44

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Fri Aug 14, 2026 00:44:45)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
6605821220,4sleep03056-21PK-Backend20:45:160
6801821140,2sleep30-21swapper/320:50:173
3056727939,14sleep00-21swapper/019:05:160
3056727939,14sleep00-21swapper/019:05:160
139099760,66rtkit-daemon0-21swapper/119:05:071
139099760,66rtkit-daemon0-21swapper/119:05:071
3067126643,15sleep30-21swapper/319:05:193
3067126643,15sleep30-21swapper/319:05:183
3133726344,13sleep20-21swapper/219:06:102
3133726344,13sleep20-21swapper/219:06:102
533752550,2sleep13056-21PK-Backend20:10:011
465892480,1sleep20-21swapper/219:50:132
415482450,9sleep013-21ksoftirqd/019:35:160
875562430,4sleep33056-21PK-Backend21:45:223
1255122400,2sleep13056-21PK-Backend23:35:121
928852330,3sleep092888-21threads22:00:230
1073112300,7sleep2341ksoftirqd/222:45:012
3177399299,18cyclictest82332-21update-manager00:25:242
3177399289,17cyclictest145270-21ntp_states00:30:192
31773992814,12cyclictest978-21snmpd23:29:582
31773992814,12cyclictest117733-21unin-run23:15:012
31773992811,15cyclictest142109-21sed00:20:232
31773992810,16cyclictest70587-21/usr/sbin/munin21:00:002
31770992819,7cyclictest883-21dbus-daemon21:50:161
1107972270,4sleep0161rcuc/022:55:010
3177399262,21cyclictest129769-21python323:45:212
3176899263,10cyclictest141209-21cpuspeed_turbos00:20:130
3177399259,14cyclictest34-21ksoftirqd/220:00:122
3177399258,16cyclictest1095-21runrttasks21:18:332
3177399258,15cyclictest67137-21cron20:50:002
3177399258,15cyclictest54199-21grep20:10:172
3177399255,6cyclictest49302-21sed19:55:212
31773992520,4cyclictest0-21swapper/220:41:012
31773992510,13cyclictest40851-21cron19:34:592
31770992521,2cyclictest261ksoftirqd/123:10:201
3176899252,9cyclictest55009-21munin-run20:15:000
3176899252,11cyclictest48563-21cat19:55:120
3176899250,11cyclictest98915-21cpu22:20:120
139099250,16rtkit-daemon0-21swapper/122:55:181
1251692250,11sleep234-21ksoftirqd/223:35:002
31778992414,8cyclictest140816-21unin-run00:20:003
3177399249,14cyclictest119430-21dump-pmu-power23:19:592
3177399249,13cyclictest34-21ksoftirqd/221:20:122
3177399248,14cyclictest137698-21cpuspeed_turbos00:10:112
3177399247,15cyclictest0-21swapper/222:05:162
3177399246,4cyclictest132791-21basename23:55:142
3177399246,16cyclictest0-21swapper/222:29:002
31773992411,12cyclictest60444-21/usr/sbin/munin20:30:212
3177399240,22cyclictest68262-21python320:50:202
3176899243,9cyclictest67762-21df_inode20:50:120
3176899242,9cyclictest83378-21df_inode21:35:140
3176899242,9cyclictest78307-21grep21:20:170
3176899242,9cyclictest35669-21cron19:20:000
3176899242,9cyclictest136722-21python300:05:250
3176899242,9cyclictest118713-21ntp_states23:15:200
3176899242,7cyclictest100580-21/usr/sbin/munin22:25:180
31768992417,6cyclictest77515-21cron21:20:000
3177399235,4cyclictest0-21swapper/222:21:342
3177399234,5cyclictest36668-21ntp_offset19:20:192
3177399234,5cyclictest128070-21sed23:40:202
3177399234,5cyclictest125798-21df_inode23:35:132
3177399233,5cyclictest72789-21cstates21:05:142
3177399232,7cyclictest1095-21runrttasks20:37:062
3177399231,9cyclictest134486-21/usr/sbin/munin00:00:132
3177399231,4cyclictest0-21swapper/221:40:122
31773992310,11cyclictest53298-21unin-run20:10:002
3177399230,8cyclictest137313-21apt00:10:002
3177399230,21cyclictest98627-21awk22:20:002
3177099230,21cyclictest107553-21apt22:45:121
3176899232,9cyclictest72989-21hddtemp_smartct21:05:170
3176899232,9cyclictest71915-21sed21:00:200
3176899232,8cyclictest85582-21memory21:40:180
3176899232,8cyclictest57361-21df20:20:120
3176899232,8cyclictest47813-21munin-run19:55:000
3176899232,8cyclictest133841-21munin-run00:00:000
31768992313,6cyclictest139320-21apt00:15:130
139099230,15rtkit-daemon0-21swapper/122:51:091
31778992220,1cyclictest0-21swapper/323:41:143
31778992220,1cyclictest0-21swapper/320:23:163
31778992218,2cyclictest3056-21PK-Backend00:35:293
31778992214,6cyclictest2636-21xdg-desktop-por23:35:203
3177899220,20cyclictest0-21swapper/321:20:213
3177399227,2cyclictest34315-21cpuspeed_turbos19:15:142
3177399226,3cyclictest79261-21sh21:29:552
3177399225,15cyclictest0-21swapper/222:48:572
3177399224,5cyclictest111164-21tr22:55:122
3177399221,4cyclictest0-21swapper/220:15:132
3177399220,8cyclictest110628-21chrt22:53:382
3177399220,7cyclictest58942-21/usr/sbin/munin20:25:122
31770992213,7cyclictest140050-21/usr/sbin/munin00:15:191
3176899222,8cyclictest55660-21hddtemp_smartct20:15:160
3176899222,8cyclictest137948-21df00:10:120
3176899222,8cyclictest127537-21df23:40:120
3176899222,6cyclictest130596-21sed23:50:010
3176899221,8cyclictest81844-21memory21:30:200
3176899221,8cyclictest64411-21irqstats20:40:150
3176899221,8cyclictest52165-21df_inode20:05:100
31768992215,4cyclictest89936-21dump-pmu-power21:55:000
3176899221,10cyclictest134400-21cstates00:00:130
139099220,15rtkit-daemon0-21swapper/120:12:221
139099220,14rtkit-daemon0-21swapper/121:55:161
139099220,14rtkit-daemon0-21swapper/119:39:011
31778992119,1cyclictest3056-21PK-Backend22:55:263
31778992119,1cyclictest3056-21PK-Backend19:40:043
31778992119,1cyclictest0-21swapper/323:48:173
31778992119,1cyclictest0-21swapper/323:00:183
31778992119,1cyclictest0-21swapper/321:35:573
31778992119,1cyclictest0-21swapper/321:05:133
31778992119,1cyclictest0-21swapper/319:39:303
31778992119,1cyclictest0-21swapper/300:26:503
31778992118,2cyclictest3056-21PK-Backend21:51:173
31778992118,2cyclictest3056-21PK-Backend19:31:383
31778992114,5cyclictest82756-21idleruntime-cro21:35:003
31778992114,5cyclictest42-21ksoftirqd/319:55:143
31778992113,6cyclictest35694-21systemd-run19:20:003
3177899210,20cyclictest0-21swapper/320:59:593
3177899210,20cyclictest0-21swapper/300:33:333
3177899210,19cyclictest3056-21PK-Backend22:45:043
3177399217,5cyclictest104430-21expr22:35:122
3177399216,13cyclictest0-21swapper/223:20:122
3177399216,13cyclictest0-21swapper/221:46:092
3177399215,3cyclictest84706-21latency_hist21:40:002
3177399215,3cyclictest31659-21turbostat.cron19:14:552
3177399213,5cyclictest57333-21expr20:20:122
31773992119,1cyclictest0-21swapper/221:34:072
31773992118,2cyclictest0-21swapper/200:19:142
3177399211,5cyclictest70833-21/usr/sbin/munin21:00:152
3177399211,3cyclictest0-21swapper/221:10:142
3177399210,6cyclictest130773-21grep23:50:122
3177399210,20cyclictest0-21swapper/223:03:502
31770992115,5cyclictest37426-21systemd-run19:25:001
31770992111,8cyclictest40894-21unin-run19:35:001
3177099210,8cyclictest868572grep21:45:141
3176899213,11cyclictest111759-21ntp_states22:55:180
3176899212,6cyclictest40847-21taskset19:34:590
3176899211,7cyclictest91597-21taskset21:59:140
3176899210,20cyclictest0-21swapper/020:35:550
3176899210,20cyclictest0-21swapper/019:20:380
139099210,15rtkit-daemon3056-21PK-Backend21:01:341
139099210,14rtkit-daemon3056-21PK-Backend21:39:381
139099210,14rtkit-daemon0-21swapper/020:30:060
816572200,4sleep13056-21PK-Backend21:30:171
558912200,4sleep13056-21PK-Backend20:15:181
3177899205,13cyclictest0-21swapper/320:00:193
3177899202,12cyclictest74329-21apt21:10:123
31778992018,1cyclictest3056-21PK-Backend22:09:163
31778992018,1cyclictest0-21swapper/323:30:333
31778992018,1cyclictest0-21swapper/323:25:473
31778992018,1cyclictest0-21swapper/300:20:083
31778992017,2cyclictest3056-21PK-Backend22:28:413
31778992017,2cyclictest0-21swapper/300:05:213
31778992015,3cyclictest66743-21users20:45:243
31778992014,4cyclictest42-21ksoftirqd/300:00:213
31778992014,4cyclictest120497-21wc23:20:203
31778992012,6cyclictest46318-21/usr/sbin/munin19:50:123
31778992011,4cyclictest84965-21apt-get21:40:033
31778992010,7cyclictest55641-21diskstats20:15:143
3177899200,1cyclictest0-21swapper/321:28:393
3177899200,1cyclictest0-21swapper/320:27:013
3177899200,1cyclictest0-21swapper/319:14:563
3177899200,19cyclictest3056-21PK-Backend22:40:583
3177899200,19cyclictest0-21swapper/323:18:093
3177899200,19cyclictest0-21swapper/300:13:593
31773992018,1cyclictest0-21swapper/219:43:352
3177399201,7cyclictest96042-21sed22:10:182
3177399200,8cyclictest146659-21/usr/sbin/munin00:35:162
3177399200,7cyclictest92349-21idleruntime22:00:172
3177399200,6cyclictest115975-21idleruntime-cro23:10:002
3177399200,6cyclictest102327-21/usr/sbin/munin22:30:242
3177399200,6cyclictest102327-21/usr/sbin/munin22:30:242
3177399200,5cyclictest44719-21grep19:45:132
31770992018,1cyclictest0-21swapper/123:05:541
31770992015,3cyclictest43622-21ntp_states19:40:181
31770992013,5cyclictest99390-21wc22:20:171
31770992012,6cyclictest86475-21basename21:45:001
3177099200,1cyclictest0-21swapper/123:01:281
3176899209,9cyclictest883-21dbus-daemon19:30:000
3176899209,3cyclictest13-21ksoftirqd/022:11:300
31768992018,1cyclictest0-21swapper/023:35:150
31768992018,1cyclictest0-21swapper/022:32:080
31768992018,1cyclictest0-21swapper/022:32:080
31768992018,1cyclictest0-21swapper/020:59:540
31768992018,1cyclictest0-21swapper/019:40:250
3176899201,1cyclictest0-21swapper/022:42:560
3176899200,6cyclictest117883-21latency_hist23:15:010
3176899200,1cyclictest0-21swapper/022:49:520
3176899200,1cyclictest0-21swapper/022:07:270
3176899200,1cyclictest0-21swapper/019:45:180
3176899200,19cyclictest0-21swapper/022:17:280
3176899200,19cyclictest0-21swapper/000:29:560
139099200,14rtkit-daemon3056-21PK-Backend00:02:491
139099200,13rtkit-daemon3056-21PK-Backend19:59:081
31778991917,1cyclictest3056-21PK-Backend22:36:533
31778991917,1cyclictest0-21swapper/323:58:253
31778991917,1cyclictest0-21swapper/319:21:243
31778991916,2cyclictest3056-21PK-Backend22:54:513
3177899190,1cyclictest3056-21PK-Backend22:00:433
3177899190,1cyclictest0-21swapper/321:15:073
3177899190,1cyclictest0-21swapper/320:42:013
3177399194,3cyclictest0-21swapper/219:25:072
3177399190,6cyclictest42047-21grep19:35:212
3177099199,9cyclictest3056-21PK-Backend21:29:591
31770991914,3cyclictest131300-21grep23:50:171
31770991911,6cyclictest883-21dbus-daemon22:15:011
31770991911,6cyclictest128631-21cron23:45:001
3176899197,9cyclictest113134-21diskstats23:00:130
31768991917,1cyclictest3056-21PK-Backend19:11:050
31768991917,1cyclictest0-21swapper/023:07:570
31768991917,1cyclictest0-21swapper/021:48:550
3176899190,6cyclictest132204-21sed23:55:010
3176899190,6cyclictest13-21ksoftirqd/020:00:020
3176899190,18cyclictest0-21swapper/000:38:350
139099190,13rtkit-daemon3056-21PK-Backend00:38:011
31778991816,1cyclictest3056-21PK-Backend23:05:123
31778991816,1cyclictest3056-21PK-Backend22:23:413
31778991816,1cyclictest0-21swapper/320:33:573
31778991815,2cyclictest3056-21PK-Backend22:32:483
31778991815,2cyclictest3056-21PK-Backend22:32:473
31778991811,5cyclictest72407-21ls21:05:013
31778991811,5cyclictest52243-21cat20:05:143
3177399184,3cyclictest0-21swapper/221:56:122
31773991816,1cyclictest0-21swapper/221:51:102
31770991816,1cyclictest3056-21PK-Backend19:10:171
31770991815,2cyclictest3056-21PK-Backend19:53:021
31770991814,2cyclictest26-21ksoftirqd/119:20:011
3176899180,7cyclictest13-21ksoftirqd/022:39:010
3176899180,6cyclictest13-21ksoftirqd/023:23:080
31778991715,1cyclictest3056-21PK-Backend22:11:173
31778991715,1cyclictest3056-21PK-Backend19:26:583
31778991715,1cyclictest0-21swapper/323:13:503
31778991715,1cyclictest0-21swapper/319:45:303
31778991712,4cyclictest54328-21ps20:10:203
31778991711,5cyclictest131376-21ntp_states23:50:193
31778991710,6cyclictest62798-21sed20:35:173
3177099179,6cyclictest74054-21cron21:10:001
31770991715,1cyclictest0-21swapper/122:33:301
31770991715,1cyclictest0-21swapper/122:33:301
31770991715,1cyclictest0-21swapper/122:01:391
31770991714,2cyclictest3056-21PK-Backend20:45:421
31770991713,3cyclictest26-21ksoftirqd/122:25:191
31770991711,5cyclictest94210-21sed22:05:171
31770991710,5cyclictest105553-21/usr/sbin/munin22:40:001
3176899174,6cyclictest13-21ksoftirqd/023:28:050
3176899173,6cyclictest13-21ksoftirqd/020:25:410
3176899172,6cyclictest13-21ksoftirqd/023:32:370
31768991715,1cyclictest0-21swapper/021:11:580
3176899170,7cyclictest13-21ksoftirqd/000:30:180
3177099166,2cyclictest141rcu_preempt20:50:171
31770991614,1cyclictest3056-21PK-Backend00:21:041
31770991612,3cyclictest57617-21grep20:20:171
31770991612,3cyclictest124894-21idleruntime-cro23:35:001
31770991612,2cyclictest26-21ksoftirqd/121:15:231
31770991612,2cyclictest26-21ksoftirqd/120:30:171
3177099161,13cyclictest38755-21awk19:25:241
31770991611,3cyclictest133167-21processes23:55:181
3177099160,8cyclictest143793-21timerandwakeup00:25:231
3176899160,6cyclictest13-21ksoftirqd/021:25:150
31778991513,1cyclictest3056-21PK-Backend21:59:143
3177099158,6cyclictest98055-21cat22:15:211
3177099158,6cyclictest44694-21ls19:45:121
3177099157,6cyclictest883-21dbus-daemon00:15:011
3177099156,2cyclictest141rcu_preempt21:22:011
3177099155,2cyclictest141rcu_preempt20:00:081
3177099154,2cyclictest141rcu_preempt00:05:481
31770991513,1cyclictest3056-21PK-Backend23:23:591
31770991513,1cyclictest3056-21PK-Backend20:56:231
31770991513,1cyclictest0-21swapper/122:40:051
31770991512,2cyclictest3056-21PK-Backend20:40:281
31770991511,2cyclictest26-21ksoftirqd/120:25:171
3177099150,14cyclictest3056-21PK-Backend23:27:101
3177099150,14cyclictest3056-21PK-Backend23:18:251
31778991411,2cyclictest3056-21PK-Backend22:20:003
3177099142,2cyclictest141rcu_preempt21:10:221
31770991410,3cyclictest3056-21PK-Backend23:45:181
31770991410,2cyclictest26-21ksoftirqd/100:30:191
3177099140,5cyclictest3056-21PK-Backend20:35:121
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional