You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-14 - 08:43

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Sun Jun 14, 2026 00:44:45)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
337123821350,2sleep20-21swapper/221:35:112
1390991120,96rtkit-daemon0-21swapper/019:08:580
342569521020,3sleep20-21swapper/200:10:182
34206762960,3sleep20-21swapper/223:55:182
33359982700,5sleep03336003-21cpuspeed_turbos19:55:130
33698542640,4sleep03056-21PK-Backend21:30:160
331904226444,14sleep30-21swapper/319:08:453
33187752630,7sleep10-21swapper/119:05:281
331891726240,15sleep20-21swapper/219:07:112
33777442400,13sleep234-21ksoftirqd/221:54:592
33765982380,4sleep10-21swapper/121:50:141
331928299387,17cyclictest1095-21runrttasks22:31:140
34228972370,3sleep20-21swapper/200:00:222
331928499361,9cyclictest34011222wc23:00:171
331928299349,13cyclictest1095-21runrttasks21:37:050
331928499301,14cyclictest3333575-21sh19:50:001
34024672290,11chrt13-21ksoftirqd/023:05:040
3319282992917,10cyclictest3332118-21apt19:45:120
3319289992811,2cyclictest141rcu_preempt21:05:002
3319289992811,2cyclictest141rcu_preempt21:04:592
3319284992813,5cyclictest3372515-21sh21:40:011
331928299288,18cyclictest3362258-21expr21:10:140
3319282992815,11cyclictest3416102-21idleruntime-cro23:45:010
3319282992813,13cyclictest3349712-21cat20:35:110
3319282992811,15cyclictest3339165-21cut20:05:000
139099280,3rtkit-daemon3056-21PK-Backend21:26:391
331928499272,9cyclictest3397049-21apt22:50:121
331928499272,9cyclictest3342773-21apt20:15:111
331928299278,17cyclictest3327138-21cat19:30:130
331928299271,12cyclictest3433988-21apt00:35:120
331928299269,15cyclictest3366165-21grep21:20:170
331928299268,16cyclictest3356972-21ls20:55:130
331928299267,17cyclictest3424583-21sed00:05:240
331928299267,17cyclictest3339451-21expr20:05:130
331928299261,4cyclictest0-21swapper/023:00:100
3319282992612,12cyclictest3373524-21sh21:40:180
3319282992612,12cyclictest3346941-21ntp_kernel_err20:25:190
139099260,17rtkit-daemon0-21swapper/121:05:171
331928499253,7cyclictest3383177-21apt22:10:091
331928299256,18cyclictest3396846-21sed22:50:000
331928299256,17cyclictest3393958-21entropy22:40:160
331928299255,19cyclictest3378201-21ls21:55:150
3319282992510,13cyclictest3322281-21memory19:15:170
34084312240,4sleep00-21swapper/023:20:180
3319289992420,3cyclictest34-21ksoftirqd/221:00:002
3319284992410,10cyclictest883-21dbus-daemon19:17:331
331928299247,16cyclictest3319776-21expr19:10:120
331928299246,17cyclictest3400336-21sed23:00:000
331928299246,17cyclictest3353592-21basename20:45:140
331928299241,3cyclictest0-21swapper/022:15:120
3319282992412,10cyclictest3342746-21/usr/sbin/munin20:15:170
3319282992410,12cyclictest3329251-21sed19:35:180
331928299240,1cyclictest0-21swapper/000:10:230
331929599230,21cyclictest3435429-21chrt00:39:543
3319289992314,7cyclictest3395673-21df22:45:132
3319284992313,7cyclictest1095-21runrttasks21:10:221
3319284992312,8cyclictest883-21dbus-daemon21:45:011
331928299238,13cyclictest3359647-21timerwakeupswit21:00:220
331928299238,13cyclictest3359647-21timerwakeupswit21:00:220
331928299238,13cyclictest2156-21gnome-shell19:21:090
331928299236,15cyclictest3341620-21missed_timers20:10:170
331928299235,17cyclictest3375591-21who21:45:220
3319282992310,11cyclictest3334213-21sh19:50:140
331929599220,20cyclictest3056-21PK-Backend22:02:243
331929599220,20cyclictest3056-21PK-Backend22:02:243
3319289992220,1cyclictest0-21swapper/220:35:532
3319289992220,1cyclictest0-21swapper/219:50:102
3319284992220,1cyclictest0-21swapper/122:28:031
3319284992220,1cyclictest0-21swapper/120:36:161
3319284992219,2cyclictest0-21swapper/120:54:551
331928299226,15cyclictest3421936-21cpuspeed00:00:120
331928299226,15cyclictest3379993-21expr22:00:130
331928299226,15cyclictest3379993-21expr22:00:130
331928299226,15cyclictest0-21swapper/023:29:080
331928299226,14cyclictest3349503-21latency_hist20:35:010
331928299221,8cyclictest3345245-21ntp_states20:20:190
3319282992217,3cyclictest3384040-21python322:10:180
331928299221,5cyclictest0-21swapper/020:40:130
331928299221,3cyclictest0-21swapper/022:05:110
331928299221,3cyclictest0-21swapper/019:25:100
3319282992210,10cyclictest3361222-21ps21:05:210
3319282992210,10cyclictest1095-21runrttasks00:20:140
331928299220,3cyclictest0-21swapper/022:50:130
139099220,14rtkit-daemon3056-21PK-Backend23:29:081
139099220,14rtkit-daemon0-21swapper/123:55:281
34033672210,4sleep126-21ksoftirqd/123:05:211
33225882210,2sleep20-21swapper/219:15:212
3319295992113,7cyclictest2164-21gdbus19:17:343
331929599210,19cyclictest3056-21PK-Backend22:53:413
331929599210,19cyclictest3056-21PK-Backend21:37:163
331929599210,19cyclictest3056-21PK-Backend20:52:133
331929599210,19cyclictest3056-21PK-Backend20:22:123
331929599210,19cyclictest3056-21PK-Backend19:55:433
331929599210,19cyclictest3056-21PK-Backend19:35:363
331929599210,19cyclictest3056-21PK-Backend19:28:183
331928999218,7cyclictest34-21ksoftirqd/222:55:002
3319289992119,1cyclictest0-21swapper/223:34:122
3319289992119,1cyclictest0-21swapper/223:27:502
3319289992119,1cyclictest0-21swapper/223:09:512
3319289992119,1cyclictest0-21swapper/222:29:362
3319289992119,1cyclictest0-21swapper/222:20:172
3319289992119,1cyclictest0-21swapper/221:43:532
3319289992119,1cyclictest0-21swapper/221:25:202
3319289992119,1cyclictest0-21swapper/221:19:062
3319289992119,1cyclictest0-21swapper/220:14:172
3319289992119,1cyclictest0-21swapper/219:41:522
3319289992113,7cyclictest3418366-21apt23:50:122
331928999210,20cyclictest0-21swapper/220:20:052
331928999210,20cyclictest0-21swapper/219:33:342
331928999210,20cyclictest0-21swapper/200:15:282
331928999210,1cyclictest3056-21PK-Backend21:10:142
3319284992119,1cyclictest3056-21PK-Backend23:34:481
3319284992119,1cyclictest3056-21PK-Backend21:23:271
3319284992119,1cyclictest0-21swapper/122:58:181
3319284992119,1cyclictest0-21swapper/122:42:021
3319284992119,1cyclictest0-21swapper/122:32:421
3319284992119,1cyclictest0-21swapper/122:05:321
3319284992111,7cyclictest3380339-21cat22:00:191
3319284992111,7cyclictest3380339-21cat22:00:181
331928499210,20cyclictest0-21swapper/119:21:401
331928499210,19cyclictest3056-21PK-Backend23:52:251
331928299217,13cyclictest0-21swapper/021:15:130
331928299217,12cyclictest3391901-21cpuspeed22:35:120
331928299217,12cyclictest0-21swapper/000:15:200
331928299213,5cyclictest3355256-21cat20:50:130
331928299211,8cyclictest3330467-21/usr/sbin/munin19:40:120
331928299211,7cyclictest3387022-21/usr/sbin/munin22:20:140
331928299211,7cyclictest3377114-21python321:50:220
331928299210,8cyclictest3388754-21diskstats22:25:140
139099210,13rtkit-daemon0-21swapper/119:10:091
3319295992018,1cyclictest3056-21PK-Backend20:32:483
3319295992017,2cyclictest3056-21PK-Backend21:52:033
3319295992014,5cyclictest3421194-21sshd23:55:303
3319295992013,6cyclictest42-21ksoftirqd/321:25:143
3319295992013,6cyclictest3408257-21sed23:20:183
3319295992013,5cyclictest3432306-21cpuspeed00:30:103
331929599200,1cyclictest3056-21PK-Backend21:46:213
331929599200,1cyclictest0-21swapper/323:39:383
331929599200,19cyclictest3056-21PK-Backend19:21:503
331929599200,19cyclictest0-21swapper/323:44:583
331929599200,19cyclictest0-21swapper/321:31:153
331929599200,19cyclictest0-21swapper/300:26:353
331929599200,18cyclictest3056-21PK-Backend23:00:373
331929599200,18cyclictest3056-21PK-Backend22:46:113
331929599200,18cyclictest3056-21PK-Backend21:08:303
3319289992018,1cyclictest0-21swapper/221:20:552
3319289992018,1cyclictest0-21swapper/219:45:372
3319289992018,1cyclictest0-21swapper/219:25:302
3319289992013,5cyclictest3414315-21sshd23:39:312
3319289992012,2cyclictest34-21ksoftirqd/219:55:162
331928999200,1cyclictest0-21swapper/222:16:252
331928999200,1cyclictest0-21swapper/221:47:452
331928999200,1cyclictest0-21swapper/221:30:352
331928999200,1cyclictest0-21swapper/200:24:412
331928999200,19cyclictest0-21swapper/222:05:022
331928999200,19cyclictest0-21swapper/220:06:062
3319284992018,1cyclictest3056-21PK-Backend23:21:431
3319284992018,1cyclictest3056-21PK-Backend23:11:091
3319284992018,1cyclictest0-21swapper/122:17:421
3319284992018,1cyclictest0-21swapper/121:57:481
3319284992018,1cyclictest0-21swapper/121:46:251
3319284992018,1cyclictest0-21swapper/120:44:401
3319284992018,1cyclictest0-21swapper/120:27:031
3319284992017,2cyclictest3056-21PK-Backend21:17:261
3319284992014,4cyclictest3345154-21grep20:20:191
3319284992012,6cyclictest883-21dbus-daemon20:14:591
3319284992011,6cyclictest3427213-21grep00:15:111
331928499200,19cyclictest3056-21PK-Backend23:19:391
331928499200,19cyclictest3056-21PK-Backend00:27:131
331928499200,19cyclictest0-21swapper/122:35:331
331928499200,19cyclictest0-21swapper/119:35:251
331928299206,12cyclictest2156-21gnome-shell21:27:090
331928299204,15cyclictest0-21swapper/023:14:340
331928299204,15cyclictest0-21swapper/000:34:340
331928299201,5cyclictest3421567-21latency_hist00:00:010
331928299200,6cyclictest3431979-21sh00:30:000
331928299200,6cyclictest3418484-21tr23:50:120
3319295991917,1cyclictest3056-21PK-Backend22:24:553
3319295991917,1cyclictest3056-21PK-Backend22:08:043
3319295991917,1cyclictest3056-21PK-Backend00:15:203
3319295991917,1cyclictest0-21swapper/323:26:053
3319295991917,1cyclictest0-21swapper/323:16:193
3319295991917,1cyclictest0-21swapper/300:20:263
3319295991914,3cyclictest3419107-21ntp_states23:50:183
3319295991914,3cyclictest3404597-21df23:10:113
331929599190,1cyclictest3056-21PK-Backend22:10:563
331929599190,1cyclictest3056-21PK-Backend20:59:033
331929599190,1cyclictest3056-21PK-Backend19:54:563
331929599190,1cyclictest0-21swapper/321:24:123
331929599190,1cyclictest0-21swapper/321:13:443
331929599190,18cyclictest3056-21PK-Backend19:41:303
331929599190,18cyclictest0-21swapper/323:05:313
331929599190,17cyclictest3412271-21anacron23:30:303
3319289991917,1cyclictest3056-21PK-Backend22:58:022
3319289991917,1cyclictest0-21swapper/223:00:052
3319289991917,1cyclictest0-21swapper/221:58:192
3319289991917,1cyclictest0-21swapper/220:47:432
3319289991917,1cyclictest0-21swapper/220:29:522
3319289991917,1cyclictest0-21swapper/219:20:232
3319289991912,2cyclictest34-21ksoftirqd/222:35:012
331928999190,1cyclictest0-21swapper/223:46:562
331928999190,1cyclictest0-21swapper/223:40:342
331928999190,1cyclictest0-21swapper/222:01:102
331928999190,1cyclictest0-21swapper/222:01:102
331928999190,1cyclictest0-21swapper/221:05:482
331928999190,1cyclictest0-21swapper/219:37:182
331928999190,1cyclictest0-21swapper/200:37:112
3319284991917,1cyclictest3056-21PK-Backend00:31:191
3319284991917,1cyclictest0-21swapper/100:38:291
3319284991916,2cyclictest3056-21PK-Backend23:47:171
3319284991916,2cyclictest0-21swapper/119:44:181
3319284991914,3cyclictest3354647-21grep20:50:011
3319284991913,4cyclictest3348641-21strings20:30:171
3319284991912,4cyclictest3340760-21cron20:10:001
3319284991911,6cyclictest3396763-21cron22:49:591
331928499190,1cyclictest3056-21PK-Backend23:42:581
331928499190,1cyclictest0-21swapper/122:20:311
331928499190,1cyclictest0-21swapper/119:51:481
331928499190,1cyclictest0-21swapper/119:25:591
331928499190,18cyclictest3056-21PK-Backend23:39:551
331928499190,18cyclictest0-21swapper/119:59:321
331928299190,6cyclictest3413313-21ls23:35:170
331928299190,5cyclictest3407181-21sh23:17:000
331929599181,6cyclictest3417341-21/usr/sbin/munin23:45:193
3319295991816,1cyclictest3056-21PK-Backend22:29:353
3319295991816,1cyclictest3056-21PK-Backend20:48:503
3319295991816,1cyclictest3056-21PK-Backend19:48:133
3319295991816,1cyclictest3056-21PK-Backend00:11:153
3319295991813,4cyclictest42-21ksoftirqd/322:37:093
3319289991816,1cyclictest0-21swapper/223:20:532
3319289991816,1cyclictest0-21swapper/222:39:572
3319289991816,1cyclictest0-21swapper/222:13:332
3319289991816,1cyclictest0-21swapper/220:44:182
3319289991816,1cyclictest0-21swapper/220:32:282
3319289991816,1cyclictest0-21swapper/200:30:022
331928499189,7cyclictest3426726-21sshd00:14:311
331928499182,13cyclictest3337573-21apt20:00:101
3319284991816,1cyclictest0-21swapper/121:03:331
3319284991816,1cyclictest0-21swapper/121:03:331
3319284991812,4cyclictest3422034-21grep00:00:121
3319284991810,4cyclictest3328538-21latency_hist19:35:011
331928299185,11cyclictest34115462chrt23:30:140
331928299180,5cyclictest3416603-21taskset23:45:090
3319295991715,1cyclictest3056-21PK-Backend22:18:263
3319295991712,3cyclictest3364322-21irqstats21:15:173
3319295991710,5cyclictest3393882-21expr22:40:143
3319289991715,1cyclictest0-21swapper/223:18:512
3319289991715,1cyclictest0-21swapper/220:53:442
3319289991713,3cyclictest978-21snmpd23:11:572
3319289991712,4cyclictest978-21snmpd20:02:562
331928499178,6cyclictest3423761-21cpuspeed_turbos00:05:141
3319284991715,1cyclictest0-21swapper/121:32:061
3319284991715,1cyclictest0-21swapper/120:56:441
34314372160,3sleep20-21swapper/200:25:212
3319295991614,1cyclictest3056-21PK-Backend21:57:443
3319295991614,1cyclictest3056-21PK-Backend20:44:373
3319295991614,1cyclictest3056-21PK-Backend20:36:123
3319295991614,1cyclictest3056-21PK-Backend19:31:163
3319295991613,2cyclictest3056-21PK-Backend21:00:533
3319295991613,2cyclictest3056-21PK-Backend21:00:533
3319295991612,3cyclictest3056-21PK-Backend20:07:513
331929599160,1cyclictest141rcu_preempt22:35:003
331928999165,3cyclictest141rcu_preempt00:05:162
331928999162,3cyclictest3056-21PK-Backend22:40:152
3319289991612,2cyclictest34-21ksoftirqd/219:10:112
3319295991513,1cyclictest3056-21PK-Backend00:05:063
3319295991513,1cyclictest0-21swapper/320:15:033
331929599149,3cyclictest3347709-21turbostat20:30:003
3319295991412,1cyclictest3056-21PK-Backend21:41:423
331929599140,2cyclictest3056-21PK-Backend20:00:113
331929599140,13cyclictest3056-21PK-Backend22:56:323
331929599140,12cyclictest3421598-40updatedb.mlocat00:00:013
331928999143,4cyclictest34-21ksoftirqd/220:16:132
139099140,3rtkit-daemon3056-21PK-Backend00:22:111
331929599130,3cyclictest3056-21PK-Backend20:10:183
331929599130,11cyclictest3056-21PK-Backend19:10:173
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional