You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-25 - 04:42

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Mon May 25, 2026 00:44:45)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
42516821260,4sleep23056-21PK-Backend20:20:202
45805121190,3sleep33056-21PK-Backend21:55:133
45805121190,3sleep33056-21PK-Backend21:55:133
45666221030,3sleep20-21swapper/221:50:182
4457432970,2sleep00-21swapper/021:20:170
139099900,52rtkit-daemon0-21swapper/319:07:323
39747128148,22sleep10-21swapper/119:05:121
39861226343,14sleep20-21swapper/219:05:472
39867226244,13sleep00-21swapper/019:06:320
4802042500,2sleep30-21swapper/323:00:013
4652392490,4sleep20-21swapper/222:15:162
4664622470,2sleep00-21swapper/022:20:120
4280952440,5chrt0-21swapper/020:30:120
39908599308,14cyclictest497520-21taskset23:49:121
39908599308,13cyclictest471260-21runrttasks22:34:071
39908599307,10cyclictest402601-21taskset19:15:291
39908599306,13cyclictest0-21swapper/123:33:301
39908599297,9cyclictest490479-21taskset23:27:471
39908599296,8cyclictest499091-21taskset23:52:091
399085992913,14cyclictest412117-21ls19:45:121
39908599288,7cyclictest0-21swapper/100:05:261
39908599287,6cyclictest499598-21apt23:55:111
39908599286,10cyclictest493960-21taskset23:37:581
399085992811,15cyclictest462784-21latency_hist22:10:001
399085992810,16cyclictest456636-21wc21:50:181
5065832270,4sleep33056-21PK-Backend00:14:553
39908599276,8cyclictest455588-21taskset21:48:071
39908599276,13cyclictest460749-21runrttasks22:02:391
399085992721,4cyclictest510108-21unin-run00:25:001
399085992715,8cyclictest427745-21/usr/sbin/munin20:30:231
399085992712,13cyclictest442042-21ls21:10:181
399085992710,15cyclictest475669-21wc22:45:181
399085992613,11cyclictest488183-21ntp_kernel_pll_23:20:171
399085992612,12cyclictest436-20systemd-journal19:23:161
139099260,19rtkit-daemon3056-21PK-Backend20:53:433
39908599259,14cyclictest465383-21ntp_kernel_err22:15:181
399085992522,2cyclictest0-21swapper/123:15:011
399085992522,2cyclictest0-21swapper/121:31:121
4074842240,3sleep20-21swapper/219:30:202
39908599246,6cyclictest430871-21runrttasks20:38:581
399085992419,4cyclictest408202-21systemd-run19:35:001
399085992417,5cyclictest477948-21taskset22:51:001
399085992416,4cyclictest434701-21apt20:50:121
399085992415,6cyclictest431238-21apt20:40:111
399085992414,6cyclictest471616-21apt22:35:121
399085992413,9cyclictest445073-21/usr/sbin/munin21:20:101
399085992412,9cyclictest411038-21/usr/sbin/munin19:40:201
139099240,15rtkit-daemon0-21swapper/222:25:112
399092992320,2cyclictest3056-21PK-Backend23:33:373
399090992321,1cyclictest0-21swapper/223:49:332
39909099230,22cyclictest0-21swapper/219:49:372
39908599237,5cyclictest436626-21cstates20:55:121
39908599236,6cyclictest441015-21runrttasks21:05:451
399085992315,5cyclictest506004-21python300:10:201
399085992314,8cyclictest452012-21cron21:40:001
399085992314,7cyclictest502440-21sshd00:00:261
399085992314,7cyclictest482242-21/usr/sbin/munin23:05:221
399085992314,7cyclictest458614-21/usr/sbin/munin21:55:201
399085992314,7cyclictest458614-21/usr/sbin/munin21:55:201
399085992313,8cyclictest512815-21ntp_states00:30:191
399085992313,8cyclictest0-21swapper/122:55:101
399085992312,9cyclictest452392-21grep21:40:111
399085992312,9cyclictest421232-21smartctl20:10:151
399085992312,9cyclictest399774-21/usr/sbin/munin19:10:171
399085992312,10cyclictest424884-21missed_timers20:20:171
399085992311,10cyclictest508372-20unin-run00:19:591
399085992311,10cyclictest468354-21cstates22:25:121
399084992313,8cyclictest448001-21switchtime21:25:210
139099230,15rtkit-daemon3056-21PK-Backend20:45:132
139099230,15rtkit-daemon0-21swapper/223:18:472
139099230,15rtkit-daemon0-21swapper/221:23:562
399092992219,2cyclictest3056-21PK-Backend23:57:293
399092992219,2cyclictest3056-21PK-Backend23:22:283
399092992219,2cyclictest3056-21PK-Backend21:22:303
399092992219,2cyclictest3056-21PK-Backend19:43:543
399092992219,1cyclictest3056-21PK-Backend19:25:033
39909299220,20cyclictest3056-21PK-Backend22:30:373
399090992220,1cyclictest0-21swapper/223:30:272
399090992220,1cyclictest0-21swapper/222:46:062
399090992220,1cyclictest0-21swapper/221:41:442
399090992220,1cyclictest0-21swapper/221:25:152
399090992220,1cyclictest0-21swapper/221:19:412
399090992220,1cyclictest0-21swapper/221:10:072
399090992220,1cyclictest0-21swapper/220:37:162
399090992220,1cyclictest0-21swapper/219:16:242
399090992220,1cyclictest0-21swapper/200:21:372
399090992220,1cyclictest0-21swapper/200:11:362
399090992211,9cyclictest501099-40mandb00:00:002
39909099220,21cyclictest0-21swapper/223:54:332
39909099220,21cyclictest0-21swapper/200:16:372
39908599228,3cyclictest404973-21apt19:25:111
39908599224,16cyclictest438331-21grep21:00:131
39908599224,16cyclictest438331-21grep21:00:131
399085992215,5cyclictest447954-21python321:25:201
399085992215,5cyclictest419848-21ntp_states20:05:201
399085992213,7cyclictest514529-21ntpq00:35:201
399085992213,7cyclictest463506-21sed22:10:161
399085992213,7cyclictest414732-21/usr/sbin/munin19:50:231
399085992212,9cyclictest467116-21/usr/sbin/munin22:20:181
399085992212,8cyclictest443476-21/usr/sbin/munin21:15:131
399085992212,8cyclictest433699-21ntpq20:45:191
399085992211,9cyclictest427504-21idleruntime-cro20:30:001
399085992211,9cyclictest417501-21hddtemp_smartct20:00:141
399084992219,2cyclictest0-21swapper/023:12:070
399084992218,2cyclictest401115-21ntp_states19:15:010
399084992215,5cyclictest515290-21munin-run00:40:010
399084992212,8cyclictest476573-21unin-run22:50:000
39908499220,20cyclictest410801-21sed19:40:160
139099220,15rtkit-daemon3056-21PK-Backend19:21:313
139099220,14rtkit-daemon0-21swapper/221:56:162
139099220,14rtkit-daemon0-21swapper/221:56:162
399092992119,1cyclictest3056-21PK-Backend23:14:303
399092992119,1cyclictest3056-21PK-Backend22:35:503
399092992119,1cyclictest3056-21PK-Backend22:07:113
399092992119,1cyclictest3056-21PK-Backend21:52:453
399092992119,1cyclictest3056-21PK-Backend21:47:103
399092992119,1cyclictest3056-21PK-Backend20:08:213
399092992119,1cyclictest3056-21PK-Backend19:18:123
399092992119,1cyclictest0-21swapper/319:37:123
399092992118,2cyclictest3056-21PK-Backend23:48:303
399092992118,2cyclictest3056-21PK-Backend22:46:593
399092992118,2cyclictest3056-21PK-Backend00:17:173
39909299210,20cyclictest3056-21PK-Backend21:08:443
39909299210,20cyclictest3056-21PK-Backend00:33:523
39909299210,20cyclictest0-21swapper/300:04:053
39909299210,19cyclictest3056-21PK-Backend19:46:043
399090992119,1cyclictest0-21swapper/223:41:362
399090992119,1cyclictest0-21swapper/223:24:392
399090992119,1cyclictest0-21swapper/223:07:362
399090992119,1cyclictest0-21swapper/222:01:112
399090992119,1cyclictest0-21swapper/221:30:492
399090992119,1cyclictest0-21swapper/220:10:252
399090992119,1cyclictest0-21swapper/219:57:552
399090992119,1cyclictest0-21swapper/219:53:222
399090992119,1cyclictest0-21swapper/219:36:592
399090992119,1cyclictest0-21swapper/219:20:302
39909099210,20cyclictest0-21swapper/223:14:522
39909099210,20cyclictest0-21swapper/219:11:452
399085992115,4cyclictest511079-21ntp_states00:25:161
399085992115,4cyclictest473474-21grep22:40:111
399085992112,7cyclictest494386-21/usr/sbin/munin23:40:111
399085992112,7cyclictest409155-21ntpq19:35:181
399085992111,8cyclictest484399-21hddtemp_smartct23:10:141
399085992111,8cyclictest423041-21awk20:15:161
399085992111,8cyclictest415681-21expr19:55:131
399085992111,8cyclictest0-21swapper/123:00:131
399084992119,1cyclictest0-21swapper/023:05:380
399084992113,6cyclictest424880-21sshd20:20:160
39908499210,2cyclictest477218-21smartctl22:50:170
39908499210,20cyclictest0-21swapper/023:54:390
39908499210,20cyclictest0-21swapper/020:35:180
39908499210,20cyclictest0-21swapper/019:27:520
139099210,15rtkit-daemon3056-21PK-Backend23:37:313
139099210,14rtkit-daemon3056-21PK-Backend21:41:073
139099210,14rtkit-daemon3056-21PK-Backend19:50:333
139099210,14rtkit-daemon3056-21PK-Backend00:23:193
139099210,13rtkit-daemon3056-21PK-Backend20:38:023
139099210,13rtkit-daemon0-21swapper/223:28:322
4416342200,3chrt3056-21PK-Backend21:10:133
399092992018,1cyclictest3056-21PK-Backend22:44:023
399092992018,1cyclictest3056-21PK-Backend22:12:253
399092992018,1cyclictest3056-21PK-Backend22:00:153
399092992018,1cyclictest3056-21PK-Backend21:34:333
399092992018,1cyclictest3056-21PK-Backend21:18:513
399092992018,1cyclictest3056-21PK-Backend00:36:303
399092992017,2cyclictest3056-21PK-Backend23:15:183
399092992017,2cyclictest3056-21PK-Backend22:26:453
39909299200,1cyclictest0-21swapper/319:55:303
39909299200,19cyclictest3056-21PK-Backend23:28:503
39909299200,19cyclictest3056-21PK-Backend22:16:173
39909299200,19cyclictest3056-21PK-Backend20:46:353
39909299200,19cyclictest3056-21PK-Backend20:02:133
39909099205,3cyclictest141rcu_preempt22:30:112
399090992018,1cyclictest0-21swapper/222:05:372
399090992018,1cyclictest0-21swapper/220:54:532
399090992018,1cyclictest0-21swapper/220:04:042
399090992018,1cyclictest0-21swapper/200:38:072
399090992017,2cyclictest3056-21PK-Backend20:25:192
39909099200,1cyclictest0-21swapper/220:41:082
39909099200,1cyclictest0-21swapper/200:29:142
39909099200,19cyclictest0-21swapper/223:38:252
39909099200,19cyclictest0-21swapper/221:39:142
39909099200,19cyclictest0-21swapper/221:05:212
39909099200,19cyclictest0-21swapper/220:33:242
39909099200,19cyclictest0-21swapper/220:05:392
39909099200,19cyclictest0-21swapper/200:33:402
399084992018,1cyclictest0-21swapper/023:02:400
399084992018,1cyclictest0-21swapper/021:51:430
399084992018,1cyclictest0-21swapper/020:52:420
399084992018,1cyclictest0-21swapper/019:54:500
399084992013,5cyclictest418265-21sshd20:01:160
39908499200,19cyclictest0-21swapper/023:59:380
39908499200,19cyclictest0-21swapper/023:43:300
39908499200,19cyclictest0-21swapper/022:31:310
39908499200,19cyclictest0-21swapper/021:16:420
39908499200,19cyclictest0-21swapper/020:15:390
139099200,14rtkit-daemon3056-21PK-Backend22:23:313
139099200,14rtkit-daemon0-21swapper/222:56:342
139099200,14rtkit-daemon0-21swapper/222:43:222
4685132190,2sleep0161rcuc/022:25:130
399092991917,1cyclictest3056-21PK-Backend23:50:063
399092991917,1cyclictest3056-21PK-Backend21:25:343
399092991917,1cyclictest3056-21PK-Backend20:33:493
399092991917,1cyclictest3056-21PK-Backend19:34:143
399092991917,1cyclictest3056-21PK-Backend19:11:023
399092991917,1cyclictest3056-21PK-Backend00:27:033
39909299190,1cyclictest3056-21PK-Backend23:08:013
39909299190,1cyclictest3056-21PK-Backend23:03:423
39909299190,1cyclictest3056-21PK-Backend21:39:543
399090991917,1cyclictest0-21swapper/222:37:272
399090991917,1cyclictest0-21swapper/221:47:532
399090991917,1cyclictest0-21swapper/220:19:112
399090991917,1cyclictest0-21swapper/219:26:452
399090991916,2cyclictest3056-21PK-Backend00:04:002
39909099190,1cyclictest0-21swapper/222:12:132
399084991917,1cyclictest0-21swapper/021:59:480
399084991917,1cyclictest0-21swapper/021:59:480
399084991917,1cyclictest0-21swapper/021:45:350
399084991917,1cyclictest0-21swapper/021:32:230
399084991917,1cyclictest0-21swapper/021:10:470
399084991917,1cyclictest0-21swapper/021:00:540
399084991917,1cyclictest0-21swapper/021:00:540
399084991917,1cyclictest0-21swapper/020:45:320
399084991917,1cyclictest0-21swapper/000:03:520
399084991912,6cyclictest883-21dbus-daemon00:08:330
399084991910,7cyclictest461952-21ntp_states22:05:200
39908499190,1cyclictest3056-21PK-Backend19:35:090
39908499190,1cyclictest0-21swapper/020:09:160
39908499190,1cyclictest0-21swapper/019:34:210
39908499190,1cyclictest0-21swapper/019:24:220
39908499190,1cyclictest0-21swapper/000:27:370
39908499190,18cyclictest0-21swapper/023:45:540
39908499190,18cyclictest0-21swapper/022:36:450
39908499190,18cyclictest0-21swapper/022:10:020
39908499190,18cyclictest0-21swapper/021:42:310
39908499190,18cyclictest0-21swapper/019:13:390
139099190,13rtkit-daemon3056-21PK-Backend20:26:243
139099190,13rtkit-daemon0-21swapper/220:58:542
399092991816,1cyclictest3056-21PK-Backend20:10:443
399092991815,2cyclictest3056-21PK-Backend20:24:373
399090991816,1cyclictest0-21swapper/200:07:242
399084991816,1cyclictest0-21swapper/020:13:150
399084991816,1cyclictest0-21swapper/019:58:350
399084991812,5cyclictest507829-21ls00:15:230
399092991715,1cyclictest3056-21PK-Backend20:42:423
399092991715,1cyclictest3056-21PK-Backend00:09:533
399092991714,2cyclictest3056-21PK-Backend23:40:463
399090991715,1cyclictest0-21swapper/223:04:382
399090991715,1cyclictest0-21swapper/222:52:492
399090991715,1cyclictest0-21swapper/222:21:322
399090991711,4cyclictest34-21ksoftirqd/221:05:002
399090991711,4cyclictest34-21ksoftirqd/221:05:002
39908499179,6cyclictest489592-21fw_packets23:25:140
399084991715,1cyclictest0-21swapper/023:30:120
399084991715,1cyclictest0-21swapper/022:56:120
399084991715,1cyclictest0-21swapper/022:15:160
399084991715,1cyclictest0-21swapper/022:01:440
399084991715,1cyclictest0-21swapper/021:37:100
399084991715,1cyclictest0-21swapper/020:26:400
399084991715,1cyclictest0-21swapper/019:46:110
399084991712,3cyclictest441358-21idleruntime-cro21:10:010
399084991710,5cyclictest512938-21python300:30:210
399084991710,5cyclictest508619-21grep00:20:120
39909299165,10cyclictest3056-21PK-Backend20:18:373
39909299160,14cyclictest436432-21apt20:55:113
399090991614,1cyclictest3056-21PK-Backend19:42:202
39908499169,5cyclictest487596-21expr23:20:120
39908499167,7cyclictest431908-21needreboot20:40:170
399084991614,1cyclictest0-21swapper/020:55:120
399084991610,4cyclictest506631-21NetworkManager00:15:000
399084991610,4cyclictest473526-21expr22:40:110
39908499158,5cyclictest485992-21expr23:15:120
39909299149,3cyclictest978-21snmpd22:50:393
399084991411,2cyclictest492647-21/usr/sbin/munin23:35:130
39909299139,3cyclictest3056-21PK-Backend21:03:363
39909299139,3cyclictest3056-21PK-Backend21:03:363
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional