You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-26 - 04:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Sun Apr 26, 2026 00:44:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
80580621010,3sleep13056-21PK-Backend21:25:231
139099790,64rtkit-daemon0-21swapper/219:05:162
75688227841,13sleep30-21swapper/319:09:073
75547927548,17sleep10-21swapper/119:05:121
75688927439,9sleep00-21swapper/019:09:120
8547322640,4sleep13056-21PK-Backend23:45:131
7651102510,3sleep3411rcuc/319:30:163
8316472360,1chrt831646-21/usr/sbin/munin22:40:120
8426482270,4chrt3056-21PK-Backend23:10:121
8621942250,3sleep13056-21PK-Backend00:05:181
75710799251,11cyclictest850365-21python323:30:223
75710299251,22cyclictest845865-21/usr/sbin/munin23:20:122
75710799240,22cyclictest803974-21rt-features21:20:203
139099240,15rtkit-daemon3056-21PK-Backend21:15:103
8614702230,5sleep00-21swapper/000:05:010
75710799231,9cyclictest844899-21python323:15:213
757107992318,3cyclictest848509-21ntp_states23:25:193
757102992314,7cyclictest883-21dbus-daemon19:34:152
757102992314,7cyclictest883-21dbus-daemon19:15:002
757094992320,2cyclictest3056-21PK-Backend23:59:241
757094992318,3cyclictest808068-21grep21:35:001
757092992321,1cyclictest0-21swapper/020:34:120
757092992318,3cyclictest819645-21/usr/sbin/munin22:05:160
757102992220,1cyclictest0-21swapper/222:45:312
757102992220,1cyclictest0-21swapper/222:45:312
757102992220,1cyclictest0-21swapper/222:06:112
757102992220,1cyclictest0-21swapper/219:35:092
757102992218,3cyclictest762746-21systemd19:25:022
757102992215,5cyclictest771386-21apt19:50:012
757102992215,5cyclictest771386-21apt19:50:002
757102992211,9cyclictest859350-21sshd23:56:422
757094992220,1cyclictest0-21swapper/122:10:411
75709499220,20cyclictest3056-21PK-Backend23:28:151
75709499220,20cyclictest3056-21PK-Backend00:31:281
757092992220,1cyclictest0-21swapper/020:36:000
757092992220,1cyclictest0-21swapper/020:23:570
757092992220,1cyclictest0-21swapper/000:34:480
139099220,16rtkit-daemon3056-21PK-Backend00:12:513
139099220,15rtkit-daemon3056-21PK-Backend00:09:463
139099220,14rtkit-daemon0-21swapper/223:47:312
75710799214,5cyclictest767272-21processes19:35:203
757107992117,2cyclictest1788620-21WebKitWebProces22:37:403
75710799210,20cyclictest0-21swapper/321:28:593
757102992119,1cyclictest0-21swapper/223:25:052
757102992119,1cyclictest0-21swapper/222:29:422
757102992119,1cyclictest0-21swapper/222:16:382
757102992119,1cyclictest0-21swapper/221:48:052
757102992119,1cyclictest0-21swapper/221:48:052
757102992119,1cyclictest0-21swapper/221:43:542
757102992119,1cyclictest0-21swapper/221:43:532
757102992119,1cyclictest0-21swapper/221:20:012
757102992119,1cyclictest0-21swapper/221:13:112
757102992119,1cyclictest0-21swapper/220:43:022
757102992119,1cyclictest0-21swapper/220:28:092
757102992119,1cyclictest0-21swapper/220:17:562
757102992119,1cyclictest0-21swapper/220:12:072
757102992119,1cyclictest0-21swapper/219:52:342
757102992119,1cyclictest0-21swapper/200:27:112
757102992119,1cyclictest0-21swapper/200:24:062
757102992113,6cyclictest776860-21apt20:05:142
757102992111,7cyclictest783521-21cron20:25:002
75710299210,20cyclictest0-21swapper/223:41:212
75710299210,20cyclictest0-21swapper/223:16:202
757094992119,1cyclictest3056-21PK-Backend22:49:211
757094992119,1cyclictest3056-21PK-Backend22:49:211
757094992119,1cyclictest3056-21PK-Backend21:18:231
757094992119,1cyclictest3056-21PK-Backend20:37:401
757094992119,1cyclictest3056-21PK-Backend19:48:051
757094992119,1cyclictest3056-21PK-Backend19:48:051
757094992119,1cyclictest0-21swapper/121:13:231
757094992118,2cyclictest3056-21PK-Backend21:37:501
757094992118,2cyclictest3056-21PK-Backend19:12:301
757094992116,3cyclictest436-20systemd-journal21:59:551
75709499210,20cyclictest0-21swapper/122:35:021
75709499210,19cyclictest3056-21PK-Backend23:36:461
75709499210,19cyclictest3056-21PK-Backend23:20:441
75709499210,19cyclictest3056-21PK-Backend22:51:101
75709499210,19cyclictest3056-21PK-Backend00:01:271
757092992119,1cyclictest3056-21PK-Backend20:48:370
757092992119,1cyclictest0-21swapper/023:41:560
757092992119,1cyclictest0-21swapper/023:29:320
757092992119,1cyclictest0-21swapper/022:20:440
757092992119,1cyclictest0-21swapper/021:44:000
757092992119,1cyclictest0-21swapper/021:44:000
757092992119,1cyclictest0-21swapper/021:26:230
757092992119,1cyclictest0-21swapper/021:10:210
757092992119,1cyclictest0-21swapper/020:41:140
757092992119,1cyclictest0-21swapper/020:27:220
757092992119,1cyclictest0-21swapper/020:13:090
757092992119,1cyclictest0-21swapper/020:00:110
757092992119,1cyclictest0-21swapper/019:55:240
757092992119,1cyclictest0-21swapper/019:51:250
757092992119,1cyclictest0-21swapper/019:49:360
757092992119,1cyclictest0-21swapper/019:49:360
757092992119,1cyclictest0-21swapper/019:41:190
757092992119,1cyclictest0-21swapper/019:35:510
757092992119,1cyclictest0-21swapper/019:16:440
757092992118,2cyclictest3056-21PK-Backend22:11:180
757092992118,2cyclictest3056-21PK-Backend21:19:330
757092992113,6cyclictest809323-21ntpq21:35:190
75709299210,20cyclictest0-21swapper/022:39:160
75709299210,20cyclictest0-21swapper/021:34:010
75709299210,20cyclictest0-21swapper/000:28:540
139099210,15rtkit-daemon0-21swapper/221:36:252
139099210,14rtkit-daemon0-21swapper/321:35:263
139099210,13rtkit-daemon0-21swapper/323:43:243
75710799205,3cyclictest832257-21missed_timers22:40:183
75710799205,13cyclictest794444-21acpi20:55:113
75710799203,15cyclictest0-21swapper/321:00:063
757107992015,3cyclictest824184-21grep22:20:003
757107992015,3cyclictest1788620-21WebKitWebProces22:57:413
757107992012,6cyclictest846351-21memory23:20:183
757107992012,6cyclictest788694-21sshd20:39:443
757107992012,6cyclictest436-20systemd-journal20:44:593
75710299209,9cyclictest769211-21timerwakeupswit19:40:212
75710299202,13cyclictest856434-21runrttasks23:50:112
757102992018,1cyclictest0-21swapper/223:12:282
757102992018,1cyclictest0-21swapper/223:01:062
757102992018,1cyclictest0-21swapper/221:09:462
757102992018,1cyclictest0-21swapper/219:19:142
757102992018,1cyclictest0-21swapper/200:15:072
757102992015,3cyclictest840560-21find23:05:122
757102992012,6cyclictest792189-21apt-get20:50:032
757102992012,5cyclictest1095-21runrttasks22:30:212
75710299200,2cyclictest0-21swapper/220:58:102
75710299200,1cyclictest0-21swapper/221:53:052
75710299200,1cyclictest0-21swapper/219:55:442
75710299200,19cyclictest0-21swapper/223:33:382
75710299200,19cyclictest0-21swapper/222:40:512
75710299200,19cyclictest0-21swapper/221:59:072
75710299200,19cyclictest0-21swapper/221:23:252
75710299200,19cyclictest0-21swapper/220:30:462
757094992018,1cyclictest3056-21PK-Backend22:59:151
757094992018,1cyclictest3056-21PK-Backend22:31:231
757094992018,1cyclictest3056-21PK-Backend22:29:341
757094992018,1cyclictest3056-21PK-Backend20:42:541
757094992018,1cyclictest3056-21PK-Backend20:19:011
757094992018,1cyclictest3056-21PK-Backend20:08:271
757094992018,1cyclictest3056-21PK-Backend19:37:581
757094992018,1cyclictest3056-21PK-Backend19:25:411
757094992018,1cyclictest3056-21PK-Backend19:23:041
757094992018,1cyclictest0-21swapper/122:01:151
757094992018,1cyclictest0-21swapper/121:05:591
757094992017,2cyclictest3056-21PK-Backend22:43:071
75709499200,1cyclictest3056-21PK-Backend23:52:281
75709499200,19cyclictest3056-21PK-Backend21:23:231
75709499200,18cyclictest3056-21PK-Backend22:20:081
75709499200,18cyclictest3056-21PK-Backend00:26:351
75709499200,18cyclictest3056-21PK-Backend00:20:401
75709499200,18cyclictest0-21swapper/119:54:411
757092992018,1cyclictest0-21swapper/022:58:290
757092992018,1cyclictest0-21swapper/021:49:480
757092992018,1cyclictest0-21swapper/021:49:480
757092992018,1cyclictest0-21swapper/019:32:120
757092992018,1cyclictest0-21swapper/019:27:460
757092992018,1cyclictest0-21swapper/019:21:450
757092992018,1cyclictest0-21swapper/000:24:000
757092992016,3cyclictest857885-21cron23:55:000
757092992013,5cyclictest778316-21latency_hist20:10:010
75709299200,1cyclictest0-21swapper/023:14:180
75709299200,1cyclictest0-21swapper/022:33:490
75709299200,19cyclictest0-21swapper/022:47:340
75709299200,19cyclictest0-21swapper/022:47:340
75709299200,19cyclictest0-21swapper/019:12:470
75709299200,19cyclictest0-21swapper/000:39:420
75709299200,19cyclictest0-21swapper/000:17:040
75709299200,18cyclictest0-21swapper/022:00:020
757107991917,1cyclictest0-21swapper/321:06:493
75710799191,5cyclictest782059-21apt20:20:113
757107991914,3cyclictest836570-21sh22:55:003
75710799191,3cyclictest760378-21python319:15:233
75710799191,16cyclictest860785-21python300:00:213
757107991911,6cyclictest856149-21cron23:50:003
757107991911,6cyclictest843182-21awk23:10:203
75710799191,16cyclictest793799-21users20:50:223
75710799191,16cyclictest0-21swapper/321:40:413
75710799191,16cyclictest0-21swapper/321:40:403
75710799190,1cyclictest0-21swapper/300:35:153
75710799190,18cyclictest0-21swapper/319:43:073
757102991917,1cyclictest0-21swapper/220:38:362
757102991914,3cyclictest978-21snmpd00:31:502
757102991913,4cyclictest804832-21/usr/sbin/munin21:25:142
757102991911,6cyclictest861759-21cat00:05:132
75710299190,1cyclictest0-21swapper/223:37:292
75710299190,1cyclictest0-21swapper/222:58:152
75710299190,1cyclictest0-21swapper/222:50:582
75710299190,1cyclictest0-21swapper/222:38:002
75710299190,1cyclictest0-21swapper/222:04:212
75710299190,1cyclictest0-21swapper/200:35:082
75710299190,18cyclictest0-21swapper/222:11:242
757094991917,1cyclictest3056-21PK-Backend23:02:531
757094991917,1cyclictest3056-21PK-Backend20:32:261
757094991917,1cyclictest3056-21PK-Backend00:37:231
757094991917,1cyclictest0-21swapper/122:08:041
757094991916,2cyclictest3056-21PK-Backend19:18:521
757094991916,1cyclictest3056-21PK-Backend23:09:221
75709499190,1cyclictest3056-21PK-Backend22:17:321
75709499190,1cyclictest3056-21PK-Backend21:45:141
75709499190,1cyclictest3056-21PK-Backend21:45:131
75709499190,1cyclictest3056-21PK-Backend19:41:361
75709499190,18cyclictest0-21swapper/121:50:131
757092991917,1cyclictest3056-21PK-Backend21:22:580
757092991917,1cyclictest0-21swapper/023:57:510
757092991917,1cyclictest0-21swapper/023:05:460
757092991917,1cyclictest0-21swapper/022:54:510
757092991917,1cyclictest0-21swapper/022:18:070
757092991917,1cyclictest0-21swapper/020:51:010
757092991917,1cyclictest0-21swapper/020:19:440
75709299190,1cyclictest3056-21PK-Backend23:15:200
75709299190,1cyclictest0-21swapper/023:02:550
75709299190,1cyclictest0-21swapper/000:14:000
75709299190,18cyclictest0-21swapper/023:30:330
139099190,13rtkit-daemon3056-21PK-Backend22:07:023
757107991816,1cyclictest0-21swapper/319:24:553
757107991814,3cyclictest777717-21sensors_temp20:05:233
757107991813,3cyclictest867054-21cnf-update-db00:20:013
757107991813,3cyclictest784543-21ntp_states20:25:193
757107991813,3cyclictest763707-21ntp_offset19:25:193
757107991812,4cyclictest770051-21cpuspeed_turbos19:45:143
757107991812,4cyclictest770051-21cpuspeed_turbos19:45:133
757107991811,6cyclictest839365-21sed23:00:183
757107991811,6cyclictest813798-21/usr/sbin/munin21:50:003
757107991811,6cyclictest813798-21/usr/sbin/munin21:50:003
75710799181,15cyclictest852096-21python323:35:203
757107991811,5cyclictest840566-21awk23:05:123
75710799181,15cyclictest785909-21diskstats20:30:143
757107991811,5cyclictest775429-21expr20:00:153
757107991811,5cyclictest436-20systemd-journal21:54:553
757107991810,6cyclictest829212-21cat22:30:223
75710799180,5cyclictest781382-21sed20:15:223
75710799180,1cyclictest0-21swapper/323:53:383
75710799180,16cyclictest869583-21ntp_states00:25:183
75710799180,16cyclictest866616-21munin-run00:20:013
75710799180,16cyclictest826391-21cnf-update-db22:25:023
75710799180,16cyclictest758685-21seq19:14:213
757102991816,1cyclictest0-21swapper/219:27:062
757102991812,4cyclictest1095-21runrttasks21:00:142
757102991811,5cyclictest824644-21grep22:20:132
757094991816,1cyclictest3056-21PK-Backend23:16:051
757094991816,1cyclictest3056-21PK-Backend20:27:271
757094991816,1cyclictest3056-21PK-Backend19:30:561
757094991816,1cyclictest3056-21PK-Backend00:19:391
757094991814,3cyclictest791739-21timerwakeupswit20:45:221
757094991811,6cyclictest774358-21sed19:55:211
757092991816,1cyclictest0-21swapper/023:36:140
757092991816,1cyclictest0-21swapper/021:54:480
75710799179,6cyclictest833662-21expr22:45:133
75710799179,6cyclictest833662-21expr22:45:133
75710799179,6cyclictest800239-21memory21:10:183
75710799179,6cyclictest42-21ksoftirqd/321:55:193
75710799175,10cyclictest42-21ksoftirqd/320:10:113
757107991715,1cyclictest0-21swapper/320:49:383
75710799171,2cyclictest821851-21python322:10:213
757107991712,4cyclictest978-21snmpd23:55:073
757107991712,4cyclictest772198-21sed19:50:173
757107991712,3cyclictest807579-21timerandwakeup21:30:233
75710799170,15cyclictest872427-21sed00:35:003
75710299178,7cyclictest859880-21systemd00:00:012
757102991715,1cyclictest3056-21PK-Backend20:45:392
757102991712,3cyclictest776033-21switchtime20:00:232
757102991711,4cyclictest808142-21readlink21:35:012
75709499179,5cyclictest864857-21/usr/sbin/munin00:15:011
757094991715,1cyclictest3056-21PK-Backend20:52:281
757094991714,2cyclictest3056-21PK-Backend23:40:381
757094991710,5cyclictest850418-21ls23:30:231
757092991715,1cyclictest3056-21PK-Backend22:25:580
757092991715,1cyclictest0-21swapper/023:48:520
757092991712,3cyclictest861841-21cpuspeed_turbos00:05:130
757092991712,3cyclictest797909-21acpi21:05:110
757092991712,3cyclictest796377-21cpuspeed_turbos21:00:140
757092991710,6cyclictest795913-21cron21:00:000
757107991614,1cyclictest3056-21PK-Backend19:59:503
757107991610,4cyclictest826051-21rm22:25:013
757102991614,1cyclictest0-21swapper/200:11:022
757094991614,1cyclictest3056-21PK-Backend20:24:021
757092991612,3cyclictest846717-21timerandwakeup23:20:210
757092991612,3cyclictest815966-21tr21:55:120
75710799150,13cyclictest0-21swapper/322:00:173
75709499158,5cyclictest797850-21wc21:05:011
75709499152,9cyclictest775129-21apt20:00:131
75709499149,4cyclictest3056-21PK-Backend21:40:541
75709499149,4cyclictest3056-21PK-Backend21:40:541
75709499149,3cyclictest7944332sleep120:55:121
757094991410,3cyclictest3056-21PK-Backend20:13:241
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional